From skip@manatee.mojam.com  Sun Dec  1 13:00:16 2002
From: skip@manatee.mojam.com (Skip Montanaro)
Date: Sun, 1 Dec 2002 07:00:16 -0600
Subject: [Python-Dev] Weekly Python Bug/Patch Summary
Message-ID: <200212011300.gB1D0GMS011844@manatee.mojam.com>

Bug/Patch Summary
-----------------

313 open / 3091 total bugs (+4)
96 open / 1810 total patches (+2)

New Bugs
--------

__all__ not in RefMan index (2002-11-24)
	http://python.org/sf/643227
typo in Object/abstract.c : ternary_op (2002-11-24)
	http://python.org/sf/643260
Pythonwin : close() on a stdout  (2002-11-25)
	http://python.org/sf/643571
New class special method lookup change (2002-11-25)
	http://python.org/sf/643841
ic module "path too long" error (2002-11-26)
	http://python.org/sf/644243
file URLs mis-handled by webbrowser (2002-11-26)
	http://python.org/sf/644246
Poor error message for augmented assign (2002-11-26)
	http://python.org/sf/644345
bdist_rpm fails when installin man pages (2002-11-27)
	http://python.org/sf/644744
for lin in file: file.tell() tells wrong (2002-11-29)
	http://python.org/sf/645594
SequenceMatcher finds suboptimal sequenc (2002-11-29)
	http://python.org/sf/645629
.extend() doc, .__doc__ too restrictive (2002-11-29)
	http://python.org/sf/645777
old UnicodeData.txt (2002-12-01)
	http://python.org/sf/646408

New Patches
-----------

Allow keyword args in dict.update() (2002-11-25)
	http://python.org/sf/643565
refactoring and documenting ModuleFinder (2002-11-25)
	http://python.org/sf/643711
Set Next Statement for Python debuggers (2002-11-25)
	http://python.org/sf/643835
Fix for os.path.expanduser for Win2000 (2002-11-25)
	http://python.org/sf/643943
Build _tkinter in setup.jaguar.py (2002-11-27)
	http://python.org/sf/644920
Allow device name in linuxaudiodev.open (2002-11-27)
	http://python.org/sf/644977
Don't FPE on FreeBSD... (2002-11-28)
	http://python.org/sf/645382
Give some clue why modules fail... (2002-11-28)
	http://python.org/sf/645383
fix buffer overrun in pmerge (2002-11-28)
	http://python.org/sf/645404
Import from Zip Archive (2002-11-29)
	http://python.org/sf/645650
better timer resolution for profile.py (2002-11-29)
	http://python.org/sf/645894

Closed Bugs
-----------

bdist builds bogus .zips (2001-03-22)
	http://python.org/sf/410541
webbrowser.py: selection of browser (2001-12-20)
	http://python.org/sf/495695
PyMapping_Keys unexported in dll (2002-08-02)
	http://python.org/sf/590207
Can't assign to __name__ or __bases__ of new class (2002-08-09)
	http://python.org/sf/593154
__getstate__() documentation is vague (2002-11-13)
	http://python.org/sf/637941
better error reporting for MRO conflict (2002-11-22)
	http://python.org/sf/642489
Webbrowser (and ic) on MacOSX failure (2002-11-23)
	http://python.org/sf/642720

Closed Patches
--------------

Import from Zip archive (2001-12-12)
	http://python.org/sf/492105
Return objects in Tkinter (2002-02-16)
	http://python.org/sf/518625
_getdefaultlocale for OS X (2002-11-03)
	http://python.org/sf/632973
Cleanup of test_strptime.py (2002-11-04)
	http://python.org/sf/633633
SimpleSets (2002-11-13)
	http://python.org/sf/638095
_strptime fixes for None locale and tz (2002-11-15)
	http://python.org/sf/639112
reST version of Lib/test/README (2002-11-20)
	http://python.org/sf/641170
time and timetz for the datetime module (2002-11-21)
	http://python.org/sf/641958
Expose PyImport_FrozenModules in imp (2002-11-22)
	http://python.org/sf/642578


From skip@pobox.com  Sun Dec  1 13:37:03 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sun, 1 Dec 2002 07:37:03 -0600
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEMPCPAB.tim_one@email.msn.com>
References: <15845.608.895210.155241@montanaro.dyndns.org>
 <LNBBLJKPBEHFEDALKOLCCEMPCPAB.tim_one@email.msn.com>
Message-ID: <15850.4223.434280.797090@montanaro.dyndns.org>

Regarding how to kill a process on Windows which was started with os.spawn*,
on the Spambayes list I asked:

    >> How would I kill that process on Windows,

Mark Hammond replied:

    This worked for me just then:

    >>> os.spawnl(os.P_NOWAIT, "f:\\windows\\notepad.exe")
    548
    >>> import win32api
    >>> win32api.TerminateProcess(548,0)

Tim also replied:

    Tim> For 2.3, I implemented something or other for the Python core;
    Tim> probably killpid();

Shouldn't one of these solutions (or some variant) appear under the veneer
of os.kill()?  I realize the semantics might not be exactly the same
(perhaps Windows solutions don't support different signals or a different
set of signals), but it seems to me that if an attempt is made to provide
the os.spawn* functions across platforms we should make a similar attempt to
provide a way to kill spawned processes as well.

Corollary question: Is there some reason the win32api stuff isn't in the
core?

Skip


From tim_one@email.msn.com  Sun Dec  1 15:43:41 2002
From: tim_one@email.msn.com (Tim Peters)
Date: Sun, 1 Dec 2002 10:43:41 -0500
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <15850.4223.434280.797090@montanaro.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEGPDAAB.tim_one@email.msn.com>

[Skip Montanaro]
> Regarding how to kill a process on Windows which was started with
> os.spawn*, on the Spambayes list I asked:
>
>     >> How would I kill that process on Windows,
>
> Mark Hammond replied:
>
>     This worked for me just then:
>
>     >>> os.spawnl(os.P_NOWAIT, "f:\\windows\\notepad.exe")
>     548
>     >>> import win32api
>     >>> win32api.TerminateProcess(548,0)
>
> Tim also replied:
>
>     Tim> For 2.3, I implemented something or other for the Python core;
>     Tim> probably killpid();

I didn't remember correctly -- I implemented waitpid() for Windows.
Processes on Windows aren't meant to be externally killed, and the docs have
dire warnings about use of TerminateProcess.  Not that people don't use it,
but it's dangerous.

> Shouldn't one of these solutions (or some variant) appear under the
> veneer of os.kill()?

Signals make little sense on Windows.

> I realize the semantics might not be exactly the same
> (perhaps Windows solutions don't support different signals

Mone -- the second argument to TerminateProcess isn't a signal number, it's
the exit code to be returned by the terminated process.

> or a different set of signals),

Nope, none.

> but it seems to me that if an attempt is made to provide
> the os.spawn* functions across platforms we should make a similar
> attempt to provide a way to kill spawned processes as well.

Be my guest <wink>.

> Corollary question: Is there some reason the win32api stuff isn't in
> the core?

I *expect* Guido would like that, provided the code fit the Python style
guide and the docs were bashed into LaTeX.  That's a massive amount of
no-fun work, and I don't expect Mark to volunteer for either.



From gward@python.net  Sun Dec  1 17:07:13 2002
From: gward@python.net (Greg Ward)
Date: Sun, 1 Dec 2002 12:07:13 -0500
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEGPDAAB.tim_one@email.msn.com>
References: <15850.4223.434280.797090@montanaro.dyndns.org> <LNBBLJKPBEHFEDALKOLCAEGPDAAB.tim_one@email.msn.com>
Message-ID: <20021201170713.GA26134@cthulhu.gerg.ca>

On 01 December 2002, Tim Peters said:
> > Shouldn't one of these solutions (or some variant) appear under the
> > veneer of os.kill()?
> 
> Signals make little sense on Windows.

And that's different from Unix somehow?  >smirk<

        Greg


From mal@lemburg.com  Sun Dec  1 20:58:16 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sun, 01 Dec 2002 21:58:16 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python	import.c,2.210,2.211
References: <DD2D1DA4-03E4-11D7-AC2C-000A27B19B96@oratrix.com>
Message-ID: <3DEA77E8.9030007@lemburg.com>

Jack Jansen wrote:
> 
> On vrijdag, nov 29, 2002, at 21:47 Europe/Amsterdam, 
> jvr@users.sourceforge.net wrote:
> 
>> Update of /cvsroot/python/python/dist/src/Python
>> In directory sc8-pr-cvs1:/tmp/cvs-serv20813/Python
>>
>> Modified Files:
>>     import.c
>> Log Message:
>> Slightly improved version of patch #642578: "Expose 
>> PyImport_FrozenModules
>> in imp". This adds two functions to the imp module: get_frozenmodules()
>> and set_frozenmodules().
> 
> 
> Something that's been bothering me about frozen modules in the classical 
> sense (i.e. those that are stored in C static data structures) is that 
> the memory used by them is gone without any chance at recovery. For big 
> frozen Python programs that are to be run on small machines this is a 
> waste of precious memory.
> 
> With modules "frozen" with set_frozenmodules you could conceivably free 
> the data again after it has been imported (similar to what MacPython-OS9 
> does with modules "frozen" in "PYC " resources).
> 
> Would that be worth the added complexity?

Allocating the data on the heap would prevent sharing the
static data between processes (provided you have multiple
of the running), so if you want to add that feature, please
make it an option and not the default.

mxCGIPython heavily relies on the fact that frozen modules
are shared between processes.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From just@letterror.com  Sun Dec  1 21:39:29 2002
From: just@letterror.com (Just van Rossum)
Date: Sun,  1 Dec 2002 22:39:29 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211
In-Reply-To: <3DEA77E8.9030007@lemburg.com>
Message-ID: <r01050400-1022-EAE2F002057511D7ADDD003065D5E7E4@[10.0.0.23]>

M.-A. Lemburg wrote:

> Allocating the data on the heap would prevent sharing the
> static data between processes (provided you have multiple
> of the running), so if you want to add that feature, please
> make it an option and not the default.

Don't worry: I think extending the freeze mechanism even further is a waste of
time; I think Jack's concerns could also be addressed with an import hook. Also,
about the checkin that that triggered Jack's remark: I'm very likely to back out
the imp.[sg]et_frozenmodules patch as what I wanted to achieve would be a much
better candidate for the zip import mechanism, which is sadly still pending.
I'll have a closer look at that and see whether I can help getting this in
before 2.3.a1.

Just


From mhammond@skippinet.com.au  Sun Dec  1 22:31:54 2002
From: mhammond@skippinet.com.au (Mark Hammond)
Date: Mon, 2 Dec 2002 09:31:54 +1100
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <15850.4223.434280.797090@montanaro.dyndns.org>
Message-ID: <LCEPIIGDJPKCOIHOBJEPMEOIHPAA.mhammond@skippinet.com.au>

> Mark Hammond replied:
>
>     This worked for me just then:
>
>     >>> os.spawnl(os.P_NOWAIT, "f:\\windows\\notepad.exe")
>     548
>     >>> import win32api
>     >>> win32api.TerminateProcess(548,0)
>
> Tim also replied:
>
>     Tim> For 2.3, I implemented something or other for the Python core;
>     Tim> probably killpid();
>
> Shouldn't one of these solutions (or some variant) appear under the veneer
> of os.kill()?  I realize the semantics might not be exactly the same

The key difference is that kill() wants a PID, while TerminateProcess wants
a handle.

Now, we *could* add a kill() (assuming it is not already there) but that
would not solve your specific problem, as you have a handle and you would
then need a platform specific way to convert it to a PID.  It seems too hard
to win.

> (perhaps Windows solutions don't support different signals or a different
> set of signals), but it seems to me that if an attempt is made to provide
> the os.spawn* functions across platforms we should make a similar
> attempt to
> provide a way to kill spawned processes as well.

Yes, a higher-level execution API would be worthwhile, especially if that
could be expressed using posix semantics as a fallback.  I am thinking of
what threading is to thread, this new API could be to the various
spawn/exec/popen functions.

> Corollary question: Is there some reason the win32api stuff isn't in the
> core?

Even if we ignore Tim's requirements ('cos I know I will <wink>) there is
still alot of work, and a *huge* amount of code.  Do we want win32api, all
of the "win32*" modules in win32all, or all of win32all.  The win32api
modules seems very useful, and it is for trivial things - but once you start
doing real things, the other win32 modules start becoming useful, and so it
goes.

On the other hand, if these extensions were in the core, I wouldn't
personally be worrying about those damn hex literal warnings <wink>.

Mark.



From skip@pobox.com  Sun Dec  1 22:45:38 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sun, 1 Dec 2002 16:45:38 -0600
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <LCEPIIGDJPKCOIHOBJEPMEOIHPAA.mhammond@skippinet.com.au>
References: <15850.4223.434280.797090@montanaro.dyndns.org>
 <LCEPIIGDJPKCOIHOBJEPMEOIHPAA.mhammond@skippinet.com.au>
Message-ID: <15850.37138.572612.258716@montanaro.dyndns.org>

    >> Shouldn't one of these solutions (or some variant) appear under the
    >> veneer of os.kill()?  I realize the semantics might not be exactly
    >> the same

    Mark> The key difference is that kill() wants a PID, while
    Mark> TerminateProcess wants a handle.

Posix.kill() wants a pid. Can't os.kill() want anything it wants?  ;-) In
particular, can't it test its arguments to see if it got a handle or a pid
then "Do The Right Thing" (tm Spike Lee I think)?  I guess I'm lobbying for
os.kill() or something like it (os.retire()?) to be the terminator for
os.spawn*, regardless what those functions return.  It seems weird to be
able to create processes on various platforms but not be able to kill them
as well.

For the python-dev types not on the spambayes list, I do have an application
in mind.  In my environment I need to tunnel SB's pop3proxy app through an
ssh connection.  Starting and stopping it works okay on my MacOSX system,
and I presume it will under other Unix-based systems.  I began thinking
about whether this would port to other platforms and noticed that os.kill()
is only supported under Unix.  It may turn out that my desire is moot if ssh
can't be run under Windows anyway.

Skip


From Wiktor Sadowski" <art@wiktorsadowski.com  Sun Dec  1 01:16:52 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Sun, 1 Dec 2002 02:16:52 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
Message-ID: <003001c298d7$550e8240$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_002D_01C298DF.B68FC6C0
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

I think Python should have a more universal import mechanism , which =
could be done
by adding a new C_CUSTOM type  to the definitions for dynamic loading =
(importdl.h) ,
a new struct _customtab to import.h and a new function to Python API =
(import.c).
Then any archive,database,internet import architecture could be =
implemented=20
in C extensions without messing with Python core.

Wiktor Sadowski
www.wiktorsadowski.com


IMPLEMENTATION
(tested on win_98)

import.h
extern DL_IMPORT(int) PyImport_AppendCustomtab(int  =
(*get_name)(char*),void (*initfunc)(char*));

struct _customtab {
 int  (*get_name)(char*);
    void (*initfunc)(char*);
 struct _customtab *next;
};



importdl.h
enum filetype {
 SEARCH_ERROR,
 PY_SOURCE,
 PY_COMPILED,
 C_EXTENSION,
 PY_RESOURCE, /* Mac only */
 PKG_DIRECTORY,
 C_BUILTIN,
 PY_FROZEN,
 C_CUSTOM,
 PY_CODERESOURCE /* Mac only */
};

import.c
struct _customtab *PyImport_Customtab;


int=20
PyImport_AppendCustomtab(int  (*get_name)(char*),void =
(*initfunc)(char*))
{
  struct _customtab *pycustomtab;
  struct _customtab *newtab=3DPyMem_NEW(struct _customtab,1);
  newtab->get_name=3Dget_name;
  newtab->initfunc=3Dinitfunc;
  newtab->next=3DNULL;

  pycustomtab=3DPyImport_Customtab;
  if (pycustomtab)
  {
  while(pycustomtab->next)
     pycustomtab=3Dpycustomtab->next;=20
  pycustomtab->next=3Dnewtab;
  }
  else
     PyImport_Customtab=3Dnewtab;
  return 1;=20
 }


static struct filedescr *
find_module(char *realname, PyObject *path, char *buf, size_t buflen,
     FILE **p_fp)
.........................................................................=
..........................
static struct filedescr fd_custom =3D {"", "", C_CUSTOM};
.........................................................................=
..........................
before find_frozen !:

if (is_custom(name)) {
   strcpy(buf, name);
   return &fd_custom;
  }

  if (find_frozen(name) !=3D NULL) {
   strcpy(buf, name);
   return &fd_frozen;
.........................................................................=
..........................
and again:
if (path =3D=3D NULL) {
  if (is_custom(name)) {
   strcpy(buf, name);
   return &fd_custom;
  }

LOAD_MODULE
static PyObject *
load_module(char *name, FILE *fp, char *buf, int type)
.........................................................................=
..........................
switch (type) {
.............................................
case C_CUSTOM:

  if (buf !=3D NULL && buf[0] !=3D '\0')
   name =3D buf;
 =20
   err =3D init_custom(name);
 =20
  if (err < 0)
   return NULL;
  if (err =3D=3D 0) {
   PyErr_Format(PyExc_ImportError,
         "Purported %s module %.200s not found",
         type =3D=3D C_BUILTIN ?
      "builtin" : type =3D=3D C_CUSTOM ? "custom":"frozen",
         name);
   return NULL;
  }
  modules =3D PyImport_GetModuleDict();
  m =3D PyDict_GetItemString(modules, name);
  if (m =3D=3D NULL) {
   PyErr_Format(
    PyExc_ImportError,
    "%s module %.200s not properly initialized",
    type =3D=3D C_BUILTIN ?
      "builtin" : type =3D=3D C_CUSTOM ? "custom":"frozen",
    name);
   return NULL;
  }
  Py_INCREF(m);
  break;

IS_CUSTOM

static int
is_custom(char *name)
{
  struct _customtab *p;

if (!PyImport_Customtab)
  return 0;

p=3DPyImport_Customtab;

while(p){
     if (p->get_name)
         if ((*p->get_name)(name))
      return 1;
     p=3Dp->next;=20
}
 return 0;
}

INIT_CUSTOM
static int
init_custom(char *name)
{
 struct _customtab *p =3D PyImport_Customtab;

 if (_PyImport_FindExtension(name, name) !=3D NULL)
  return 1;

 while(p) {
     if (p->get_name){
   if ((*p->get_name)(name)) {
    if (p->initfunc =3D=3D NULL) {
     PyErr_Format(PyExc_ImportError,
      "Cannot re-init internal module %.200s",
      name);
     return -1;
    }
    if (Py_VerboseFlag)
     PySys_WriteStderr("import %s # builtin\n", name);
    (*p->initfunc)(name);
    if (PyErr_Occurred())
     return -1;
    if (_PyImport_FixupExtension(name, name) =3D=3D NULL)
     return -1;
    return 1;
   }
  }
  p=3Dp->next;
 }
 return 0;
}






------=_NextPart_000_002D_01C298DF.B68FC6C0
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV>I think Python should have a more universal import mechanism , =
which could=20
be done</DIV>
<DIV>by adding a new C_CUSTOM type&nbsp; to the definitions for dynamic =
loading=20
(importdl.h) ,</DIV>
<DIV>a new struct _customtab to import.h and a new function =
to&nbsp;Python API=20
(import.c).</DIV>
<DIV>Then any&nbsp;archive,database,internet import&nbsp;architecture =
could be=20
implemented </DIV>
<DIV>in C extensions without messing with Python core.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Wiktor Sadowski</DIV>
<DIV><A =
href=3D"http://www.wiktorsadowski.com">www.wiktorsadowski.com</A></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>IMPLEMENTATION</DIV>
<DIV>(tested on win_98)</DIV>
<DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>import.h</DIV>
<DIV>extern DL_IMPORT(int) PyImport_AppendCustomtab(int&nbsp;=20
(*get_name)(char*),void (*initfunc)(char*));</DIV>
<DIV>&nbsp;</DIV>
<DIV>struct _customtab {<BR>&nbsp;int&nbsp;=20
(*get_name)(char*);<BR>&nbsp;&nbsp;&nbsp; void=20
(*initfunc)(char*);<BR>&nbsp;struct _customtab *next;<BR>};</DIV></DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV>importdl.h</DIV>
<DIV>enum filetype=20
{<BR>&nbsp;SEARCH_ERROR,<BR>&nbsp;PY_SOURCE,<BR>&nbsp;PY_COMPILED,<BR>&nb=
sp;C_EXTENSION,<BR>&nbsp;PY_RESOURCE,=20
/* Mac only=20
*/<BR>&nbsp;PKG_DIRECTORY,<BR>&nbsp;C_BUILTIN,<BR>&nbsp;PY_FROZEN,<BR>&nb=
sp;C_CUSTOM,<BR>&nbsp;PY_CODERESOURCE=20
/* Mac only */<BR>};</DIV>
<DIV>&nbsp;</DIV>
<DIV>import.c</DIV>
<DIV>struct _customtab *PyImport_Customtab;</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV>int <BR>PyImport_AppendCustomtab(int&nbsp; (*get_name)(char*),void=20
(*initfunc)(char*))<BR>{<BR>&nbsp; struct _customtab =
*pycustomtab;<BR>&nbsp;=20
struct _customtab *newtab=3DPyMem_NEW(struct _customtab,1);<BR>&nbsp;=20
newtab-&gt;get_name=3Dget_name;<BR>&nbsp; =
newtab-&gt;initfunc=3Dinitfunc;<BR>&nbsp;=20
newtab-&gt;next=3DNULL;</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp; pycustomtab=3DPyImport_Customtab;</DIV>
<DIV>&nbsp; if (pycustomtab)<BR>&nbsp; {<BR>&nbsp;=20
while(pycustomtab-&gt;next)<BR>&nbsp;&nbsp;&nbsp;&nbsp;=20
pycustomtab=3Dpycustomtab-&gt;next; <BR>&nbsp;=20
pycustomtab-&gt;next=3Dnewtab;<BR>&nbsp; }<BR>&nbsp;=20
else<BR>&nbsp;&nbsp;&nbsp;&nbsp; PyImport_Customtab=3Dnewtab;</DIV>
<DIV>&nbsp; return 1; <BR>&nbsp;}</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV>static struct filedescr *<BR>find_module(char *realname, PyObject =
*path,=20
char *buf, size_t buflen,<BR>&nbsp;&nbsp;&nbsp;&nbsp; FILE **p_fp)</DIV>
<DIV>....................................................................=
...............................</DIV>
<DIV>static struct filedescr fd_custom =3D {"", "", C_CUSTOM};</DIV>
<DIV>....................................................................=
...............................</DIV>
<DIV>before find_frozen !:</DIV>
<DIV>&nbsp;</DIV>
<DIV>if (is_custom(name)) {<BR>&nbsp;&nbsp;&nbsp;strcpy(buf,=20
name);<BR>&nbsp;&nbsp;&nbsp;return =
&amp;fd_custom;<BR>&nbsp;&nbsp;}</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;&nbsp;if (find_frozen(name) !=3D NULL)=20
{<BR>&nbsp;&nbsp;&nbsp;strcpy(buf, name);<BR>&nbsp;&nbsp;&nbsp;return=20
&amp;fd_frozen;</DIV>
<DIV>....................................................................=
...............................</DIV>
<DIV>and again:</DIV>
<DIV>if (path =3D=3D NULL) {</DIV>
<DIV>&nbsp; if (is_custom(name)) {<BR>&nbsp;&nbsp;&nbsp;strcpy(buf,=20
name);<BR>&nbsp;&nbsp;&nbsp;return =
&amp;fd_custom;<BR>&nbsp;&nbsp;}</DIV>
<DIV>&nbsp;</DIV>
<DIV>LOAD_MODULE</DIV>
<DIV>static PyObject *<BR>load_module(char *name, FILE *fp, char *buf, =
int=20
type)</DIV>
<DIV>....................................................................=
...............................</DIV>
<DIV>switch (type) {</DIV>
<DIV>.............................................</DIV>
<DIV>case C_CUSTOM:</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;&nbsp;if (buf !=3D NULL &amp;&amp; buf[0] !=3D=20
'\0')<BR>&nbsp;&nbsp;&nbsp;name =3D =
buf;<BR>&nbsp;&nbsp;<BR>&nbsp;&nbsp;&nbsp;err=20
=3D init_custom(name);<BR>&nbsp;&nbsp;<BR>&nbsp;&nbsp;if (err &lt;=20
0)<BR>&nbsp;&nbsp;&nbsp;return NULL;<BR>&nbsp;&nbsp;if (err =3D=3D 0)=20
{<BR>&nbsp;&nbsp;&nbsp;PyErr_Format(PyExc_ImportError,<BR>&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=20
"Purported %s module %.200s not=20
found",<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type =3D=3D =
C_BUILTIN=20
?<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"builtin" : type =3D=3D =
C_CUSTOM ?=20
"custom":"frozen",<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=20
name);<BR>&nbsp;&nbsp;&nbsp;return =
NULL;<BR>&nbsp;&nbsp;}<BR>&nbsp;&nbsp;modules=20
=3D PyImport_GetModuleDict();<BR>&nbsp;&nbsp;m =3D =
PyDict_GetItemString(modules,=20
name);<BR>&nbsp;&nbsp;if (m =3D=3D NULL)=20
{<BR>&nbsp;&nbsp;&nbsp;PyErr_Format(<BR>&nbsp;&nbsp;&nbsp;&nbsp;PyExc_Imp=
ortError,<BR>&nbsp;&nbsp;&nbsp;&nbsp;"%s=20
module %.200s not properly initialized",<BR>&nbsp;&nbsp;&nbsp;&nbsp;type =
=3D=3D=20
C_BUILTIN ?<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"builtin" : type =
=3D=3D C_CUSTOM=20
?=20
"custom":"frozen",<BR>&nbsp;&nbsp;&nbsp;&nbsp;name);<BR>&nbsp;&nbsp;&nbsp=
;return=20
NULL;<BR>&nbsp;&nbsp;}<BR>&nbsp;&nbsp;Py_INCREF(m);<BR>&nbsp;&nbsp;break;=
</DIV>
<DIV>&nbsp;</DIV>
<DIV>IS_CUSTOM</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>static int<BR>is_custom(char *name)<BR>{<BR>&nbsp; struct =
_customtab=20
*p;</DIV>
<DIV>&nbsp;</DIV>
<DIV>if (!PyImport_Customtab)<BR>&nbsp; return 0;</DIV>
<DIV>&nbsp;</DIV>
<DIV>p=3DPyImport_Customtab;</DIV>
<DIV>&nbsp;</DIV>
<DIV>while(p){<BR>&nbsp;&nbsp; &nbsp; if=20
(p-&gt;get_name)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if=20
((*p-&gt;get_name)(name))<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return=20
1;<BR>&nbsp;&nbsp;&nbsp;&nbsp; p=3Dp-&gt;next; <BR>}<BR>&nbsp;return =
0;<BR>}</DIV>
<DIV>&nbsp;</DIV>
<DIV>INIT_CUSTOM</DIV>
<DIV>static int<BR>init_custom(char *name)<BR>{<BR>&nbsp;struct =
_customtab *p =3D=20
PyImport_Customtab;</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;if (_PyImport_FindExtension(name, name) !=3D=20
NULL)<BR>&nbsp;&nbsp;return 1;</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;while(p) {<BR>&nbsp;&nbsp;&nbsp; &nbsp;if=20
(p-&gt;get_name){<BR>&nbsp;&nbsp;&nbsp;if ((*p-&gt;get_name)(name))=20
{<BR>&nbsp;&nbsp;&nbsp;&nbsp;if (p-&gt;initfunc =3D=3D NULL)=20
{<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PyErr_Format(PyExc_ImportError,<BR>&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"Cannot=20
re-init internal module=20
%.200s",<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;name);<BR>&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;return=20
-1;<BR>&nbsp;&nbsp;&nbsp;&nbsp;}<BR>&nbsp;&nbsp;&nbsp;&nbsp;if=20
(Py_VerboseFlag)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PySys_WriteStderr("impo=
rt %s #=20
builtin\n",=20
name);<BR>&nbsp;&nbsp;&nbsp;&nbsp;(*p-&gt;initfunc)(name);<BR>&nbsp;&nbsp=
;&nbsp;&nbsp;if=20
(PyErr_Occurred())<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return=20
-1;<BR>&nbsp;&nbsp;&nbsp;&nbsp;if (_PyImport_FixupExtension(name, name) =
=3D=3D=20
NULL)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return=20
-1;<BR>&nbsp;&nbsp;&nbsp;&nbsp;return=20
1;<BR>&nbsp;&nbsp;&nbsp;}<BR>&nbsp;&nbsp;}<BR>&nbsp;=20
p=3Dp-&gt;next;<BR>&nbsp;}<BR>&nbsp;return 0;<BR>}<BR></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2><FONT face=3DArial =
size=3D2></FONT></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>&nbsp;</DIV></FONT></BODY></HTML>

------=_NextPart_000_002D_01C298DF.B68FC6C0--



From greg@cosc.canterbury.ac.nz  Mon Dec  2 03:23:21 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Mon, 02 Dec 2002 16:23:21 +1300 (NZDT)
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211
In-Reply-To: <DD2D1DA4-03E4-11D7-AC2C-000A27B19B96@oratrix.com>
Message-ID: <200212020323.gB23NLC21006@kuku.cosc.canterbury.ac.nz>

Jack Jansen <Jack.Jansen@oratrix.com>:

> Something that's been bothering me about frozen modules in the
> classical sense (i.e. those that are stored in C static data
> structures) is that the memory used by them is gone without any
> chance at recovery.

With virtual memory, that's probably not a problem, since
it'll get paged out after it's been used.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From barry@python.org  Mon Dec  2 04:39:13 2002
From: barry@python.org (Barry A. Warsaw)
Date: Sun, 1 Dec 2002 23:39:13 -0500
Subject: [Python-Dev] Mac OSX issues
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15850.58353.356475.182137@gargle.gargle.HOWL>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried
    GvR> other values, but the default of 512 (KB) is insufficient.  I
    GvR> know the README file explains this, but now that I've
    GvR> experienced this myself, I wonder if we shouldn't hack main()
    GvR> to increase the stack size to 2 MB, inside an #ifdef darwin
    GvR> or something like that.

+1, but I'll defer to Jack.  We discussed this a while back (for
Py2.2.2 IIRC), which is when I updated the README bit.  FWIW,
Pipermail (in Mailman) suffers the same fate and contains the
following bit of code.

-Barry

-------------------- snip snip --------------------
# MacOSX has a default stack size that is too small for deeply recursive
# regular expressions.  We see this as crashes in the Python test suite when
# running test_re.py and test_sre.py.  The fix is to set the stack limit to
# 2048; the general recommendation is to do in the shell before running the
# test suite.  But that's inconvenient for a daemon like the qrunner.
#
# AFAIK, this problem only affects the archiver, so we're adding this work
# around to this file (it'll get imported by the bundled pipermail or by the
# bin/arch script.  We also only do this on darwin, a.k.a. MacOSX.
if sys.platform == 'darwin':
    try:
        import resource
    except ImportError:
        pass
    else:
        soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
        newsoft = min(hard, max(soft, 1024*2048))
        resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))


From brett@python.org  Mon Dec  2 04:54:52 2002
From: brett@python.org (Brett Cannon)
Date: Sun, 1 Dec 2002 20:54:52 -0800 (PST)
Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30
Message-ID: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU>

Here  is the rough draft of  the Summary.  Guys have about a day to point
out its flaws.


++++++++++++++++++++++++++++++++++++++++++++++++++
python-dev Summary, 2002-11-16 through 2002-11-30
++++++++++++++++++++++++++++++++++++++++++++++++++

======================
Summary Announcements
======================
Nothing to report to speak of.

====================
`bsddb3 imported`__
====================
__ http://mail.python.org/pipermail/python-dev/2002-November/030247.html

Martin v. Loewis merged bsddb3 3.4.0 into CVS under the name ``bsddb``.
The old ``bsddb`` module is now no longer compiled by default; if it  does
get compiled, though, it ends up with the name ``bsddb185``.  Barry Warsaw
also requested that the extensive testing suite be incorporated and "run
it only with a regrtest -u option".

Martin wasn't sure how Barry wanted them incorporated, though, since there
are multiple files to the test and most testing suites in the stdlib are a
single file.  Barry suggested that the testing files be put in a directory
with the package and that test_bsddb.py just call the tests in that
directory, much like how the email package does it.  They were integrated
and some errors and warnings were found that are being dealt with.

It was also agreed upon that development will be moved over to Python so
as to keep the module in Python sync'ed up properly and to keep poor
Martin from having to import the files into Python's CVS constantly.

=======================
`Licensing question`__
=======================
__ http://mail.python.org/pipermail/python-dev/2002-November/030256.html

David Abrahams asked about  a licensing issue with Boost.Python_ (it is a
free library that "enables seamless interoperability between C++" and
Python) and it's modified Python.h_ file that it uses.  Originally there
was no license at the top of that file, but that does not work for some
corporations using Boost.  So David stuck his own license at the top and
asked if this is the right thing to do.

Guido asked him to provide the PSF_ license_ at the top of the file and to
mention what changes he made.  The copyright had been added to the file
for Python 2.2.2.

.. _Boost.Python: http://www.boost.org/libs/python/doc/
.. _Python.h:
.. _PSF: http://www.python.org/psf/
.. _license: http://www.python.org/2.2.2/license.html

=========================
`Re: PyNumber_Check()`__
=========================
__ http://mail.python.org/pipermail/python-dev/2002-November/030236.html

M.A. Lemburg noticed that PyNumber_Check()'s semantics on what will cause
it to return had changed.  He  asked if it should check whether one of
"nb_int, nb_long, nb_float is available (in addition to the tp_as_number
slot)".  Guido responded that he would like to see it deprecated.  We got
a history lesson of how PyNumber_Check() was written "when the presence or
absence of the as_number "extension" to the type object was thought to be
useful".  Regardless, Guido said that testing like this does  not prove
something is a "number" and if you wanted to test this way you could do it
yourself.

In response, MAL said that perhaps PyNumber_Check() should be changed so
that it  returned true for something that is "usable as input to float(),
int() or long()".  Guido said that would be fine "as long as we all agree
that that's *exactly* what they check for, and as long as we agree that
there may be overlapping areas" for the various Py*_Check() functions.
Guido later said testing for nb_int, nb_long, and nb_float was fine.

===================================================
`Plea: can modulefinder.py move to the library?`__
===================================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030240.html

Just van Rossum wanted to move Freeze's modulefinder.py_ into the stdlib
so that  it can be distributed  with binary releases.  In case you don't
know what modulefinder.py does, it attempts to find all pure Python module
dependencies for a pure Python module.  In other words, it checks what the
module imports and if it is a Python file, and if it is, records that; it
repeats this for all modules it finds, creating a listing of modules
needed for the module to run.

Guido said that the module needed some work before it could be considered;
it had ``print`` statements that were unneeded outside of Freeze and it
had no documentation.  Just agreed that the documentation needed to be
done.  As for the ``print`` statements, though, they only come out when
``debug`` is set to true; by default it  is  false.  Guido said that was
fine and agreed with the removal of the Windows-specific ``print``
statements.

Thomas Heller later said in `another thread`_ that `patch #643711`_ was
opened primarily for him and Just to do work in but that everyone was
invited to help out.

.. _another thread:
http://mail.python.org/pipermail/python-dev/2002-November/030445.html
.. _modulefinder.py:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Tools/freeze/modulefinder.py
.. _patch #643711: http://www.python.org/sf/643711

============================
`Dictionary Foolishness?`__
============================
__ http://mail.python.org/pipermail/python-dev/2002-November/030282.html

Raymond Hettinger suggested having "dictionaries support the repetition"
to allow one to create a dictionary with enough space as specified by the
repetition::

	>>> [0] * n   # allocate an n-length list
	>>> {} * n    # allocate an n-element dictionary

Aahz recalled that dictionaries are resized upon adding to a dictionary
and they could theoretically grow smaller.  That would seem to possibly
limit the usefulness of this idea.  Guido then voted -1 (practically a
death wish for an idea unless people clamor for it) saying that it relied
too much on "arbitrary magic by side effect".  He said  if people *really*
wanted this a method could be proposed.

=============================
`dict() enhancement idea?`__
=============================
__ http://mail.python.org/pipermail/python-dev/2002-November/030290.html

Just van Rossum suggested overloading the dictionary constructor so that
arguments that went to ``**kwargs`` would be used to create the dictionary
(this can be seen in the "`Python Cookbook`_" as recipe 1.2 or online at
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52313).  This is
desired because that means cleaner code for creating dicts::

	>>> dict(pigs='!fly', birds='fly')

Barry commented that he  liked it and had something similar in his code
for Mailman_ .  Thomas Heller voted +1 for it and also said that he used
the idiom.  Raymond Hettinger and myself also voted +1 for it.

.. _Python Cookbook:
.. _Mailman:

===========================================
`Yet another string formatting proposal`__
===========================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030304.html

Oren Tirosh proposed something (read the title to figure out what).  He
proposed the following syntax::

	>>> "\(a) + \(b) = \(a+b)\n"
   	>>> r"\(a) + \(b) = \(a+b)\n".cook()

The advantages, according to Oren, are that it would not require
introducing the use of a new symbol like ``$``, nor a new string prefix
nor a new method.  The ``.cook()`` method would be used to evaluate raw
strings at a later point; it would draw arguments from the local and
global namespace.  The biggest drawback was unfamiliarity for programmers.

Frederik Lundh pointed out that ``\(`` is "commonly used to escape
parentheses in regular expression strings" (Effbot wrote re_ , so he
should know).  Oren then said that curly braces could (and pretty will) be
used instead.

Michael Chermside likes this design idea, but  thinks the name for
``.cook()`` is not that great.  Oren was going for a name that tied into
"raw".  Michael suggested the name ``.sub()`` to build off of the two PEPs
already in existence covering string formatting (`PEP 215`_ and `PEP 292`_
).

.. _re: http://www.python.org/doc/current/lib/module-re.html
.. _PEP 215: http://www.python.org/peps/pep-0215.html
.. _PEP 292: http://www.python.org/peps/pep-0292.html

=====================
`Expect in python`__
=====================
__ http://mail.python.org/pipermail/python-dev/2002-November/030313.html

Eric Raymond proposed adding pexpect_ to the stdlib when it reaches
version 1 (it is currently at 0.94).  His thought was that having
functionality like Expect_ would be a boon for Python and use for system
administration.  Eric said  he had been using the module and had no
problems with it (Prabhu Ramachandran also said it had worked for him).

David Ascher said that he would like to see a more abstract API to allow
it work for things other than character streams.  He also would like to
see something work better on Windows.  Eric said that he would not want to
hold up this for hopes of getting something better since it already works
well for what it does.

But it appears that the creator of pexpect is more than willing to help
maintain the module if it makes it into the stdlib.

Zach Weinberg said that he would be willing to put some work into making
the pty_ module more portable since pexpect does its thing using pty.

.. _pexpect: http://pexpect.sf.net/
.. _Expect: http://expect.nist.gov/
.. _pty: http://www.python.org/doc/current/lib/module-pty.html

===================================
`PEP 288:  Generator Attributes`__
===================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030321.html

Raymond Hettinger has revised `PEP 288`_ with a new proposal on how to
pass things into a generator that has already started.  He has asked for
comments on the changes, so let him know what you think.

.. _PEP 288: http://www.python.org/peps/pep-0288.html

===============================================
`PyMem_MALLOC (was [Python-Dev] Snake farm)`__
===============================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030333.html

Continuation of
http://mail.python.org/pipermail/python-dev/2002-November/029853.html

There was a possible issue with ``PyMem_MALLOC()`` that Marc Recht had
discovered on FreeBSD.  It eventually was tracked down to FreeBSD's
implementation of ``malloc()``: ``malloc(0)`` always return 0x800.  M.A.
Lemburg suggested changing a test in the configure script to try to catch
when a platform returned an address for ``malloc(0)`` and treat it just
like when it  would return ``NULL`` (``NULL`` can't be blindly returned
since that would signal a memory error; returning ``NULL`` in a C
extension signals an error).  Marc came back with news that C99 says that
this is legitimate behavior for ``malloc()`` so this could possibly affect
other platforms.

Marc suggested that ``PyMem_MALLOC()`` just be redefined to ``n ?
malloc(n) : NULL``.  Problem is that the ``NULL`` issue mentioned above
comes into play with this solution.  Tim Peters suggested either
``malloc(n || 1)`` or ``malloc(n ? n : 1)`` (the former being a Python
idiom that doesn't cut it in C).  he does not want to mess with the
configure scripts since they have "proven itself too brittle too many
times".  Tim wanted a way to prevent ever calling the function with 0, but
Guido couldn't see any way of doing that without an extra jump.

The committed solution is ``malloc((n) ? (n) : 1)``.  Easier to just waste
one byte then have to deal with the special casing of passing 0.  The
extra test was not really a worry since no measurable performance reported
by Tim.  Besides, Tim pointed out "this is ideal for a conditional-move
instruction, and more architectures are growing that".

====================================================
`Half-baked proposal: * (and **?) in assignments`__
====================================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030349.html

Gareth McCaughan suggested cutting down one of the separations between
parameter passing and assignment by allowing assignment to use arbitrary
argument lists::

	>>> a,b,*c = 1,2,3,4,5  # c == (3, 4, 5)
	>>> year, month, day, *dummy = time.localtime()

I argued that I didn't like the slightly cluttered look on the left-hand
side (LHS) of the assignment.  Martin v. Loewis and I basically ended up
saying we wanted to keep assignments clear and concise and that this would
not help to keep that.  Steve Holden basically ended up agreeing.

Brian Quinlin, Patrick O'Brien, Nathan Clegg and Timothy Delany liked the
idea.  The biggest argument in support was that it would allow for a more
functional programming style (and that obviously can be good or bad
depending on your P.O.V.; I say bad  =)::

	>>> car,*cdr = [head, t1, t2, t3]  # car == head, cdr == (t1, t2,
t3)

In case you don't have functional programming (especially Lisp/Scheme)
experience, the basic data structure in Lisp-like language is a list and
the most common way to manipulate those lists is with the functions
``car`` and ``cdr``.  ``car`` returns the "head", or front, of the list;
``cdr`` returns the "tail", or everything but the head, of the list.  This
allows for simple recursion since you just pass the ``cdr`` of a list on
the recursive call after having dealt with the head of the list.

There was also the suggestion of allowing the arbitrary assignment
variable to be anywhere in the list of assignment variables::

	>>> a,*b,c = 1,2,3,4,5  # a == 1, b == (2, 3, 4), c == 5

To prove that this was not really needed I wrote a function that took in
an iterable and the number of variables to assign to and then returned the
proper number iterations on the iterator and then the iterator as the last
thing returned.  Alex Martelli of course improved upon it (and also
continued to correct my slightly incorrect statements)::

	def peel(iterable, arg_cnt=1):
		"""Return ``arg_cnt`` values from iterator of ``iterable``
and then the iterator itself."""
		iterator = iter(iterable)
		for num in xrange(arg_cnt):
			yield iterator.next()
		yield iterator

The idea of a module for the stdlib containing iterator helper functions
was suggested by Alex.  One is in progress by Raymond Hettinger.

Armin Rigo suggested having iterators become a type.  That was quickly
shot down, although having the suggested iterator helper module contain a
class that could be subclassed by iterators was received with positive
comments.

The thread ended very quickly after Guido said that he didn't think "that
there's a sufficient need to add new syntax".

===================================
`from tuples to immutable dicts`__
===================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030358.html

Armin Rigo said that he would like to have an immutable type that acted
like a dictionary; basically like a struct from C.  Martin v. Loewis
agreed on the need, but opposed the idea of adding another built-in type
or syntax for such a type; that left something for the stdlib.  Martin
suggested something like::

	>>> struct_seq(name, doc, n_in_sequence, (fields))

where ``field`` is a bunch of (name, doc) tuples.  What would be returned
would be a "thing [that] would be similar to os.stat_result: you [can]
call it with the mandatory fields in sequence, and can call it with the
optional fields by keyword argument".

Armin didn't like it since it went against his initial proposal "which was
to have a lightweight and declaration-less way to build structures".  He
basically ended up suggesting something along the lines of tuples with
keyword arguments.  Martin didn't like it since he didn't see a great use
for it.

In the end Armin said to just drop the idea.

============================================
`urllib performance issue on FreeBSD 4.x`__
============================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030391.html

Andrew MacIntyre brought a thread on python-list to python-dev's attention
about urllib performance compared to wget  (wget is used to download web
sites and files).  Apparently a patch was applied to Python 2.2.2 that
made the used socket be unbuffered instead of using the system default.
The question became why this was done.

The answer (thanks to Martin v. Loewis) was to prevent deadlock.
Apparently under HTTP 1.1 a server can keep a connection open while
waiting for the next command.  If the connection was buffered it would
block until it read enough to fill the buffer which may never come.

Frederik Lundh suggested that a subclass or option be available that
allowed the choosing of unbuffered or not.  Andrew said he would put it on
his todo list.

=====================================
`test failures on Debian unstable`__
=====================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030384.html

Failures on the build of Debian's unstable version of Python led to a
discussion about how modules are skipped in the testing suite.
`Lib/test/regrtest.py`_ keeps a list of tests that are expected to be
skipped on various platforms.  Martin v. Loewis doesn't like it because
tests such as for the bz2 module are attempted regardless of whether the
bz2 library is even installed and yet it is expected to succeed on Linux.
Martin summarized that "For many of the tests that are somtimes skipped,
knowing the system does not tell you whether the test will should
rightfully be skipped, on that system" since tests are skipped often
because a module was not there that needed to be imported for the test.

Tim Peters, on the other hand, likes it.  Since he maintains the Windows
distribution from PythonLabs he likes it since it lets him know when new
things have been added to Python and might need to be excluded from the
Windows distro.  Neil (who pointed out the Debian problems) was able to
recognize that the tests that failed were meant to pass under Linux.  Tim
admitted he only cared about keeping the mechanism for Windows; he could
care less if it is removed for Linux.

Patrick O'Brien chimed in (with Aahz supporting) that the feature is handy
since you can easily find out libraries you are missing that you could
potentially install.

Guido stepped in and suggested setting up a mechanism that would allow an
external table in a file to be used when present instead of the default
list of tests to skip.  Don't think anyone has stepped up to implement
this.

.. _Lib/test/regrtest.py:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/test/regrtest.py

===================================================================
`Currently baking idea for dict.sequpdate(iterable, value=True)`__
===================================================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030431.html

Raymond Hettinger presented "a write-up for a proposed dictionary update
method".  It basically took an iterable and added keys based on the values
returned by the iterator with a value as passed in and used  for all new
keys.  The rationale was to have a fast way to be able to do membership
testing using dict's ``__contains__`` or removing duplicates by creating a
dict and then outputting the keys using the aptly named ``.keys()``.


Previous objecctions to something like this were about the dict
constructor and the ``sets`` module.  The ones about the constructor are
dealt with by making this a method.  The latter was argued against by
saying that the ``sets`` module is slow.  Frederik Lundh brought up that
we really don't need multiple ways of doing the same thing.  Just van
Rossum agreed and said this killed the idea for him.  Guido chimed in and
said that the ``sets`` module was to help solidify the sets API so that at
some point it could be coded in C.

To address the speed complaint Guido suggested limiting the ``sets``
module initially to make it faster so that the type won't be held back or
unutilized because of its speed.  Tim Peters spoke up, though, and said
that the spambayes_ project used ``sets`` and he didn't have any
complaints.  But when major membership testing was needed a dict was used.
And Tim pointed out that in order for any C sets code to be fast it would
have to directly use dict's C ``__contains__`` code.

What this method should return was brought up by Just.  Some thought
``None`` since ``.update()`` returns that.  Others said ``True``.  Guido
said ``None`` since ``True`` should only be used  when something is
explicitly true.

Making it a class method was also suggested by Just as an easy way to make
it like a constructor.  Raymond agreed and changed his proprosal as well
as to have the method be named ``.fromseq()``.  But then Walter Dorwald
said ``.fromkeyseq()`` should be used  since there "is another constructor
that creates the dict from a sequence of items".  Guido voted +1 on that
idea.

.. _spambayes: http://spambayes.sf.net/

======================================
`Re: release22-maint branch broken`__
======================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030330.html

Tim Rice discovered that trying to build Python from a directory other
then where the source was did not work for the Python 2.2.* CVS.  It was
all eventually solved and fixed in the CVS branch.  I am mentioning it
here in case someone reading this had a similar issue.

================================
`Dictionary evaluation order`__
================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030458.html

Gustavo Niemeyer asked about how to handle code like ``{f1():f2(), f3():
f4()}`` and its execution order as pointed out by `bug #448679`_ .  As it
stood it evaluated in the order of f2, f1, f4, f3.  Apparently Guido once
upon a time considered this a bug.

But Guido mentioned that left-to-right evaluation is not always wanted
since ``a = {}; a[f1()] = f2()`` would want f2 to evaluate first.  He
asked what Jython did.

Finn Bock said that Jython went f1, f2, f3, f4.  In that case Guido didn't
see any reason to fix it.  But Tim Peters brought up the point that the
bug was more about the lack of specifics on this in the documentation.
Gustavo said he would make the code fix along with patches to the docs.

.. _bug #448679: http://www.python.org/sf/448679

===========================
`int/long FutureWarning`__
===========================
__ http://mail.python.org/pipermail/python-dev/2002-November/030520.html

Mark Hammond asked how the upcoming change in Python 2.4 of hex/oct
constants will affect his C extension code and something like
``PyArg_ParseTuple()`` (this function takes arguments passed to something
and breaks it up into its individual parts since all arguments are passed
as tuples in C code).  In case you don't know about the warnings, Python
2.3 warns you that code  like ``SOMETHING = 0x80000000`` could have a
different meaning in Python 2.4; most likely it will be treated as a
positive long.  You can currently get rid of the warnings by changing the
constant into a long by tacking on a ``L`` to the end of the number.

Martin v. Loewis that if Mark appended the ``L`` to his constants that it
would not work for an ``i`` argument for ``PyArg_ParseTuple()``.  But
Guido stepped up and said that there will be no issue since Python will be
changed so that Mark's code will accept the constant as a positive long.
This caused Guido to wonder if the warning could be changed to some other
warning that is not normally printed out.

Guido then mentioned that he has "long promised a set of new format codes
for PyArg_ParseTuple() to specify taking the lower N bits (for N in 8, 16,
32, 64) and throwing the rest away, without range checks".  "If
someone else can get to this first, that would be great".  So someone be
nice to Guido and do this for him.  =)

Either way no specific resolution has been reached.  As of right now you
can just  live with the warnings, supress the warnings, or change your
constants to longs and hope you are not passing into a C extension
function that wants an int.

==========================================
`assigning to new-style-class.__name__`__
==========================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030475.html

Michael Hudson has been working on `patch #635933`_ to allow for
assignment to ``__name__`` and ``__bases__`` for new-style classes (this
was all so that ``__name__`` would handle nested classes properly to allow
for proper pickling; that thread was called `metaclass insanity`_ ).  He
ran into a slight issue with dealing with assigning to ``__name__``.  To
get it working, Michael wanted to treat heap and non-heap types
differently.  For non-heap types Michael wanted to "everything in tp_name
up to the first dot is __module__, the rest is __name__".  For non-heap
types, he wanted to have ``__module__`` as "always __dict__['__module__'],
__name__ is always tp_name (or rather ((etype*)type)->name)".  And as for
the issue of if someone is crazy enough to delete the dict key of
``__module__``, Michael said Python wouldn't crash but you probably would
not like the outcome of running code.  =)

Guido responded saying that Michael's proposal was acceptable.

But then there was an issue with ``.mro()`` after the bases had been
rearranged.  Michael worried about what to do when there was a conflict
down the intheritence tree.  He thought reverting back to the way things
were if there was an issue was best.  This would require keeping around
copies of the previous states until the changes propogated all the way
through.

Samuele Pedroni stepped in to try to answer this question (Samuele rewrote
the MRO code recently and is directly mentioned in `C3 implementation`_ ).
He came up with a possible case where there could be a possible order
disagreement if two of the bases of a class had the same bases but one had
the order swapped compared to the other (so C has bases of (A, B) and D
has bases of (A, B) as well and E had bases (C, D); if C's bases became
(B, A), E now has an order disagreement).  He suggested that "the mros of
the subclasses should be computed lazily when needed (e.g. on the first -
after the changes - dispatch), although this may produce inconsistences
and errors at odd times".

Michael showed that his solution would catch the problem.  But he did not
like the idea of lazily evaluating; he wanted a more restrictive solution
since this is a new thing.  Michael stated that what he wanted this for
was to "to swap out one class for another -- making instances of the old
class instances of the new class, which was possible and making subclasses
of the old subclasses of the new, which wasn't".  It also turned out
neither  APL nor Dylan allow this kind of thing so Michael is breaking new
ground.

Samuele asked about when the classes had solid bases (i.e., only a single
superclass such as ``object``).  Michael said it  would handled with no
problem.

.. _C3 implementation:
http://mail.python.org/pipermail/python-dev/2002-October/029230.html
.. _patch #635933: http://www.python.org/sf/635933
.. _metaclass insanity:
http://mail.python.org/pipermail/python-dev/2002-November/029872.html

=====================
`Classmethod Help`__
=====================
__ http://mail.python.org/pipermail/python-dev/2002-November/030452.html

Raymond Hettinger emailed the list because  Guido said that the few people
in the world who understand descriptors for C code are on the list.  The
main reason I am mentioning the thread here, though, is because Armin Rigo
gave the answer that "There are METH_CLASS and METH_STATIC flags that you
can set in the tp_methods table".

You also learn thanks to Guido that you should only use
``PyErr_BadInternalCall()`` when you know that a "bad argment must have
been created by a broken piece of C code".




From tim.one@comcast.net  Mon Dec  2 05:44:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 02 Dec 2002 00:44:30 -0500
Subject: [Python-Dev] RE: how to kill process on Windows started with
 os.spawn?
In-Reply-To: <15850.37138.572612.258716@montanaro.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEIMDAAB.tim.one@comcast.net>

[Skip Montanaro]
> Posix.kill() wants a pid. Can't os.kill() want anything it wants?  ;-)

Yes.  On Windows it wants a handle, because that's what spawn*() returns on
Windows.  But, again, Windows processes aren't intended to be killed
externally.

> In particular, can't it test its arguments to see if it got a handle
> or a pid then "Do The Right Thing" (tm Spike Lee I think)?

No, process handles and process ids are both just "little integers" on
Windows.

> ...
> For the python-dev types not on the spambayes list, I do have an
> application in mind.  In my environment I need to tunnel SB's
> pop3proxy app through an ssh connection.  Starting and stopping
> it works okay on my MacOSX system, and I presume it will under other
> Unix-based systems.  I began thinking about whether this would port
> to other platforms and noticed that os.kill() is only supported under
> Unix.

Why do you need to kill a process externally?  For example, why can't you
make "please stop now" a part of the protocol, so that a process can
terminate itself gracefully when told to?

> It may turn out that my desire is moot if ssh can't be run under
> Windows anyway.

http://www.python.org/dev/devfaq.html#w1



From martin@v.loewis.de  Mon Dec  2 09:25:22 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 02 Dec 2002 10:25:22 +0100
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIMDAAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCEEIMDAAB.tim.one@comcast.net>
Message-ID: <m3wums4vst.fsf@mira.informatik.hu-berlin.de>

Tim Peters <tim.one@comcast.net> writes:

> Yes.  On Windows it wants a handle, because that's what spawn*()
> returns on Windows.  But, again, Windows processes aren't intended
> to be killed externally.

That sounds like FUD. Why is it that processes aren't intended to be
killed? Because the assassin can't know what state the process is in,
so the process may not complete correctly?

This is a general problem with killing, not specific to Windows, and
it never stopped a killer.

Or can you crash the operating system or the killer process by
terminating some other process?

> Why do you need to kill a process externally?  For example, why can't you
> make "please stop now" a part of the protocol, so that a process can
> terminate itself gracefully when told to?

Because the process did not respond to the protocol. Should I kill it
anyway and send a bug report to Microsoft?

Regards,
Martin


From Jack.Jansen@cwi.nl  Mon Dec  2 09:59:51 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Mon, 2 Dec 2002 10:59:51 +0100
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: <15850.58353.356475.182137@gargle.gargle.HOWL>
Message-ID: <CCD9B99A-05DC-11D7-8B12-0030655234CE@cwi.nl>

On Monday, Dec 2, 2002, at 05:39 Europe/Amsterdam, Barry A. Warsaw 
wrote:

>
>>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:
>
>     GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried
>     GvR> other values, but the default of 512 (KB) is insufficient.  I
>     GvR> know the README file explains this, but now that I've
>     GvR> experienced this myself, I wonder if we shouldn't hack main()
>     GvR> to increase the stack size to 2 MB, inside an #ifdef darwin
>     GvR> or something like that.
>
> +1, but I'll defer to Jack.  We discussed this a while back (for
> Py2.2.2 IIRC), which is when I updated the README bit.  FWIW,
> Pipermail (in Mailman) suffers the same fate and contains the
> following bit of code.

I'm +0 on this. The reason I'm not +1 is that 2MB is just as arbitrary 
as 1MB, and
it's only picked because it'll make the current set of tests pass. And 
the
current set of tests doesn't pass on MacOSX because another equally 
arbitrary
number in test_re happens to create a stack >= 1MB but <= 2MB. I would
be +1 on just changing the test to fit in a 1MB stack.

That is, unless there are subtleties in the test that I missed, and 
there really
is a reason for the specific choice of the number 50000.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From guido@python.org  Mon Dec  2 10:01:12 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 02 Dec 2002 05:01:12 -0500
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: Your message of "Sun, 01 Dec 2002 23:39:13 EST."
 <15850.58353.356475.182137@gargle.gargle.HOWL>
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net>
 <15850.58353.356475.182137@gargle.gargle.HOWL>
Message-ID: <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net>

>     GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried
>     GvR> other values, but the default of 512 (KB) is insufficient.  I
>     GvR> know the README file explains this, but now that I've
>     GvR> experienced this myself, I wonder if we shouldn't hack main()
>     GvR> to increase the stack size to 2 MB, inside an #ifdef darwin
>     GvR> or something like that.

[Barry]
> +1, but I'll defer to Jack.  We discussed this a while back (for
> Py2.2.2 IIRC), which is when I updated the README bit.  FWIW,
> Pipermail (in Mailman) suffers the same fate and contains the
> following bit of code.

Thanks!  I've added your code to regrtest.py.  I'm reluctant to do
this in PyMain(), because I don't know whether this adds to Python's
resource usage; the re tests are pretty extreme.

I'm also considering disabling test_largefile on Mac OSX unless a -u
option is given.

I'm still seeing errors in test_socket; today (on a different network)
I got this:

test test_socket failed -- Traceback (most recent call last):
  File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes
    ip = socket.gethostbyname(hostname)
gaierror: (7, 'No address associated with nodename')

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Jack.Jansen@cwi.nl  Mon Dec  2 10:06:44 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Mon, 2 Dec 2002 11:06:44 +0100
Subject: [Python-Dev] PyArg_Parse (was: int/long FutureWarning)
Message-ID: <C3066C56-05DD-11D7-8B12-0030655234CE@cwi.nl>

Martin:
> Jack Jansen <Jack.Jansen@oratrix.com> writes:
>
>> How about taking a completely different angle on this matter, and
>> looking at PyArg_Parse itself? If we can keep PyArg_Parse 100%
>> backward compatible (which would mean that its "i" format would take
>> any IntObject or LongObject between -1e31 and 1e32-1) and introduce a
>> new (preferred) way to parse arguments that not only does the right
>> thing by being expressive enough to make a difference between
>> "currency integers" and "bitmap integers", but also cleans up the
>> incredible amount of cruft that PyArg_Parse has accumulated over the
>> years?
>
> I had a similar idea, so I'd encourage you to spell out your proposal
> in more detail, or even in an implementation.
> My idea was to provide a ParseTuple wrapper, [...]

> Those of you needing to support older Python releases could
>
> #define PyArg_ParseTupleLenient PyArg_ParseTuple
>
> in your distribution, or provide other appropriate wrappers.

Unfortunately this wouldn't work if the distributions are binary 
distributions.

But, aside from that, I think I would want to go much further than this
_if_ I put time in redesigning PyArg_Parse. I would first like to take
inventory of all the problems there are with PyArg_Parse, and then see
whether we can design something that will solve most or all of these
issues, without being overly complex in everyday use.

And before I embark on that journey I would first like to have a group
of people willing to put effort into this, plus the go-ahead of Guido
(there's little point in designing a new mechanism if there is no
chance of it being adopted as the general case, especially if this new
mechanism may need a new PyMethodDef flag or some such thing).

As a kickoff, here are some of my gripes about PyArg_Parse.

1. The format chars are arcane and without any logic. There is no logic
to signed/unsigned type specifiers, some modifiers are suffixes (s#), 
some
are different format chars (s versus z), some are prefixes (e).
Everyone knows a basic set of 5 or 6 and has to look the rest up. Some
types have special shortcuts without a clear rationale (String and
Unicode are the only types to have an "O!", typename shortcut in the
form of "S" and "U").

2. There is conversion information interspersed with the argument list,
for instance with O!, O& or es formats. This makes it very difficult to
represent or build an argument parsing format in Python (this is
worsened because some of the C objects in the argument list, such as
the O& routine pointers, have no Python equivalent). And representing
an argument list parser in Python is something that you need if you
want to do dynamic wrapping of any kind (calldll, PyObjC, etc).

3. There is no way to create new, temporary objects during argument
parsing, because there is no corresponding "release" call and no way to
make the caller release new objects. Having temporary objects would
make conversion a lot easier. Unicode and strings are the first types
that come to mind, but there are probably others.

4. PyArg_ParseTupleAndKeywords makes the situation even worse. Each
argument now has *three* different "index positions", the real index in
the keyword list, a modified one in the format string (ignore all
non-alphabetic chars and "e") and a third one in the argument list
(ignore all extraneous arguments corresponding to es or O& or
what-have-you).
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From guido@python.org  Mon Dec  2 10:30:41 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 02 Dec 2002 05:30:41 -0500
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: Your message of "Mon, 02 Dec 2002 10:59:51 +0100."
 <CCD9B99A-05DC-11D7-8B12-0030655234CE@cwi.nl>
References: <CCD9B99A-05DC-11D7-8B12-0030655234CE@cwi.nl>
Message-ID: <200212021030.gB2AUfR24830@pcp02138704pcs.reston01.va.comcast.net>

[Jack]
(BTW all your mail comes poorly folded like shown here.  What's up
with that?)

> I'm +0 on this. The reason I'm not +1 is that 2MB is just as arbitrary 
> as 1MB, and
> it's only picked because it'll make the current set of tests pass. And 
> the
> current set of tests doesn't pass on MacOSX because another equally 
> arbitrary
> number in test_re happens to create a stack >= 1MB but <= 2MB. I would
> be +1 on just changing the test to fit in a 1MB stack.
> 
> That is, unless there are subtleties in the test that I missed, and 
> there really
> is a reason for the specific choice of the number 50000.

I vaguely recall that the number 50000 is somewhat historically
significant, but I don't remember why.  But it works on all other
platforms I know.

Do you know if setting the stack limit actually allocates that much
memory for the stack in the process in Mac OSX, or does it only
reserve VM space (like on traditional Unix)?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec  2 10:33:07 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 02 Dec 2002 05:33:07 -0500
Subject: [Python-Dev] PyArg_Parse (was: int/long FutureWarning)
In-Reply-To: Your message of "Mon, 02 Dec 2002 11:06:44 +0100."
 <C3066C56-05DD-11D7-8B12-0030655234CE@cwi.nl>
References: <C3066C56-05DD-11D7-8B12-0030655234CE@cwi.nl>
Message-ID: <200212021033.gB2AX7E24846@pcp02138704pcs.reston01.va.comcast.net>

> But, aside from that, I think I would want to go much further than this
> _if_ I put time in redesigning PyArg_Parse. I would first like to take
> inventory of all the problems there are with PyArg_Parse, and then see
> whether we can design something that will solve most or all of these
> issues, without being overly complex in everyday use.
> 
> And before I embark on that journey I would first like to have a group
> of people willing to put effort into this, plus the go-ahead of Guido
> (there's little point in designing a new mechanism if there is no
> chance of it being adopted as the general case, especially if this new
> mechanism may need a new PyMethodDef flag or some such thing).

+1 on a redesign of PyArg_Parse.  Though I don't want to hold up the
2.3a1 release.

> As a kickoff, here are some of my gripes about PyArg_Parse.

(I can't spend too much time not paying attention to the Zope3
sprintathon, so I'll try to read these later.  Chances are I agree.
Surely the format characters are a big mess.)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec  2 10:37:23 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 02 Dec 2002 05:37:23 -0500
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: Your message of "Sun, 01 Dec 2002 02:16:52 +0100."
 <003001c298d7$550e8240$0201010a@wiktor>
References: <003001c298d7$550e8240$0201010a@wiktor>
Message-ID: <200212021037.gB2AbNh24921@pcp02138704pcs.reston01.va.comcast.net>

> I think Python should have a more universal import mechanism , which
> could be done by adding a new C_CUSTOM type to the definitions for
> dynamic loading (importdl.h) , a new struct _customtab to import.h
> and a new function to Python API (import.c).  Then any
> archive,database,internet import architecture could be implemented
> in C extensions without messing with Python core.
> 
> Wiktor Sadowski

No time to review in detail, but +1 on the concept.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mwh@python.net  Mon Dec  2 11:21:41 2002
From: mwh@python.net (Michael Hudson)
Date: 02 Dec 2002 11:21:41 +0000
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: Jack Jansen's message of "Mon, 2 Dec 2002 10:59:51 +0100"
References: <CCD9B99A-05DC-11D7-8B12-0030655234CE@cwi.nl>
Message-ID: <2mr8d0k6nu.fsf@starship.python.net>

Jack Jansen <Jack.Jansen@cwi.nl> writes:

> I'm +0 on this. The reason I'm not +1 is that 2MB is just as
> arbitrary as 1MB,

Not quite; it's also the stack size passed to pthreads IIRC.  So after
ulimit -s 2048 all threads get the same stack size (I think).

Cheers,
M.

-- 
42. You can measure a programmer's perspective by noting his
    attitude on the continuing vitality of FORTRAN.
  -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html


From just@letterror.com  Mon Dec  2 11:37:37 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  2 Dec 2002 12:37:37 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <003001c298d7$550e8240$0201010a@wiktor>
Message-ID: <r01050400-1022-8708B8C205EA11D7ADDD003065D5E7E4@[10.0.0.23]>

Wiktor Sadowski wrote:

> I think Python should have a more universal import mechanism , which
> could be done by adding a new C_CUSTOM type  to the definitions for
> dynamic loading (importdl.h) , a new struct _customtab to import.h
> and a new function to Python API (import.c). Then any
> archive,database,internet import architecture could be implemented in
> C extensions without messing with Python core.

(Could you please post a normal diff of your patch? It looks very interesting.)

I think I agree with this concept as well. Yesterday I tried to review the
latest incarnation of the "import from zip file" patch and ran away screaming:
it's way too intrusive on import.c, touches too many files and is otherwise
simply incomprehensible to me.

I've been doing some thinking about the import mechanism, too, and my current
gripe is about the __import__ hook: I find it utterly useless that the
implementer of the hook is supposed to deal with sys.modules. What I wish for is
a hook that only gets invoked if the module isn't already in sys.modules.

Here's a quick idea. Say we add a new dynamic variable to sys named
"import_hooks". It's a list that is empty by default, but may contain import
hooks. If an import occurs, sys.modules is checked first, and *only* if the
module isn't yet loaded, something like this will be done:

    for hook in sys.import_hooks:
        mod = hook(absolutemodulename, parentpackage_or_None)
        if mod is not None:
            return mod
    return builtin_import_hook(absolutemodulename, parentpackage_or_None)

(Hm, instead of calling builtin_import_hook() explicitly, it could also be part
of the default sys.import_hooks list. Yeah, that's better, as it allows one to
completely disable builtin import.)

For an import from within a package, the mechanism is possibly invoked twice.
For example if a submodule of Foo imports Baz, the mechanism is called once with
"Foo.Baz" and if that fails (returns None), it's called again with "Baz".

I think this will make writing import hooks a) more attractive and b) far less
brittle, since there's so much less to implement.

A new-style import hook could use sys.path if that makes sense for it, but for
(say) a zip file importer it makes more sense (to me at least) to just add a
hook per zip file. No need to allow non-string types on sys.path.

I'm not entirely sure yet how my idea and yours could be integrated, but I think
it's work a look.

Just


From Jack.Jansen@cwi.nl  Mon Dec  2 12:38:23 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Mon, 2 Dec 2002 13:38:23 +0100
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: <200212021030.gB2AUfR24830@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <F2891280-05F2-11D7-8B12-0030655234CE@cwi.nl>

On Monday, Dec 2, 2002, at 11:30 Europe/Amsterdam, Guido van Rossum 
wrote:
> I vaguely recall that the number 50000 is somewhat historically
> significant, but I don't remember why.  But it works on all other
> platforms I know.

Except all other platforms with small stacks (MacOS9, WinCE, probably 
more).

> Do you know if setting the stack limit actually allocates that much
> memory for the stack in the process in Mac OSX, or does it only
> reserve VM space (like on traditional Unix)?

I don't know, but I can't imagine it does anything more than reserve VM 
space.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From Jack.Jansen@cwi.nl  Mon Dec  2 12:45:38 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Mon, 2 Dec 2002 13:45:38 +0100
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: <2mr8d0k6nu.fsf@starship.python.net>
Message-ID: <F5CF11FE-05F3-11D7-8B12-0030655234CE@cwi.nl>

On Monday, Dec 2, 2002, at 12:21 Europe/Amsterdam, Michael Hudson wrote:

> Jack Jansen <Jack.Jansen@cwi.nl> writes:
>
>> I'm +0 on this. The reason I'm not +1 is that 2MB is just as
>> arbitrary as 1MB,
>
> Not quite; it's also the stack size passed to pthreads IIRC.  So after
> ulimit -s 2048 all threads get the same stack size (I think).

On MacOSX the pthread default stack size is 512K, which is the same as 
the default stack ulimit.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From barry@python.org  Mon Dec  2 13:15:00 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 2 Dec 2002 08:15:00 -0500
Subject: [Python-Dev] Mac OSX issues
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net>
 <15850.58353.356475.182137@gargle.gargle.HOWL>
 <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15851.23764.693884.646123@gargle.gargle.HOWL>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> [Barry]
    >> +1, but I'll defer to Jack.  We discussed this a while back
    >> (for Py2.2.2 IIRC), which is when I updated the README bit.
    >> FWIW, Pipermail (in Mailman) suffers the same fate and contains
    >> the following bit of code.

    GvR> Thanks!  I've added your code to regrtest.py.  I'm reluctant
    GvR> to do this in PyMain(), because I don't know whether this
    GvR> adds to Python's resource usage; the re tests are pretty
    GvR> extreme.

Cool, works for me.

    GvR> I'm also considering disabling test_largefile on Mac OSX
    GvR> unless a -u option is given.

    GvR> I'm still seeing errors in test_socket; today (on a different
    GvR> network) I got this:

    GvR> test test_socket failed -- Traceback (most recent call last):
    GvR>   File "/Users/guido/python/src/Lib/test/test_socket.py",
    GvR> line 223, in testHostnameRes ip =
    GvR> socket.gethostbyname(hostname)
    GvR> gaierror: (7, 'No address associated with nodename')

I'll be booting up OSX tonight, so I'll try doing a cvsup and running
the tests locally.

-Barry


From barry@python.org  Mon Dec  2 13:32:04 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 2 Dec 2002 08:32:04 -0500
Subject: [Python-Dev] PyArg_Parse (was: int/long FutureWarning)
References: <C3066C56-05DD-11D7-8B12-0030655234CE@cwi.nl>
Message-ID: <15851.24788.810331.792541@gargle.gargle.HOWL>

>>>>> "JJ" == Jack Jansen <Jack.Jansen@cwi.nl> writes:

    JJ> And before I embark on that journey I would first like to have
    JJ> a group of people willing to put effort into this, plus the
    JJ> go-ahead of Guido (there's little point in designing a new
    JJ> mechanism if there is no chance of it being adopted as the
    JJ> general case, especially if this new mechanism may need a new
    JJ> PyMethodDef flag or some such thing).

You might not want to wait that long, but it would be a good topic for
a Pycon sprint.

-Barry


From fredrik@pythonware.com  Mon Dec  2 15:01:14 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 2 Dec 2002 16:01:14 +0100
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>
Message-ID: <077101c29a13$a9b78cf0$0900a8c0@spiff>

brett wrote:

> those programs where deadlock would occur do not lose or gain anything
> since they couldn't run on a platform without ``thread`` anyway.

    try:
        import threading
    except ImportError:
        threading =3D None # use non-threaded behaviour

</F>



From guido@python.org  Mon Dec  2 15:36:53 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 02 Dec 2002 10:36:53 -0500
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: Your message of "Mon, 02 Dec 2002 16:01:14 +0100."
 <077101c29a13$a9b78cf0$0900a8c0@spiff>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>
 <077101c29a13$a9b78cf0$0900a8c0@spiff>
Message-ID: <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>

>     try:
>         import threading
>     except ImportError:
>         threading = None # use non-threaded behaviour

But then you'd have to sprinkle the rest of the code with
"if threading is not None" tests.  Brett's dummy_thread is an attempt
to reduce this clutter.  I've been known to write a mini-version of
dummy_thread.py in code to minimize the clutter; it seems useful to
generalize this.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec  2 16:23:06 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 02 Dec 2002 11:23:06 -0500
Subject: [Python-Dev] Sleepycat Berkeley DB on OSX
Message-ID: <200212021623.gB2GN6g21364@pcp02138704pcs.reston01.va.comcast.net>

When running test_bsddb3 on Mac OSX, using BSDDB 4.0.14, I get
tracebacks like these:

Exception in thread writer 0:
Traceback (most recent call last):
  File "/Users/guido/python/src/Lib/threading.py", line 416, in __bootstrap
    self.run()
  File "/Users/guido/python/src/Lib/threading.py", line 404, in run
    apply(self.__target, self.__args, self.__kwargs)
  File "/Users/guido/python/src/Lib/bsddb/test/test_thread.py", line 115, in writerThread
    dbutils.DeadlockWrap(d.put, key, self.makeData(key), max_retries=12)
  File "/Users/guido/python/src/Lib/bsddb/dbutils.py", line 61, in DeadlockWrap
    except _db.DBLockDeadlockError:
NameError: global name '_db' is not defined

and these:

Exception in thread reader 8:
Traceback (most recent call last):
  File "/Users/guido/python/src/Lib/threading.py", line 416, in __bootstrap
    self.run()
  File "/Users/guido/python/src/Lib/threading.py", line 404, in run
    apply(self.__target, self.__args, self.__kwargs)
  File "/Users/guido/python/src/Lib/bsddb/test/test_thread.py", line 140, in readerThread
    c = d.cursor()
DBNoMemoryError: (12, 'Cannot allocate memory -- Lock table is out of available locks')

Any clues???

I don't get these on Linux, although there I get a failure too:

test test_bsddb3 failed -- errors occurred; run in verbose mode for details

Frankly, verbose mode waas too verbose to bother. :-(

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@python.org  Mon Dec  2 16:37:07 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 2 Dec 2002 11:37:07 -0500
Subject: [Python-Dev] Sleepycat Berkeley DB on OSX
References: <200212021623.gB2GN6g21364@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15851.35891.468176.894108@gargle.gargle.HOWL>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> Any clues???

    GvR> I don't get these on Linux, although there I get a failure
    GvR> too:

    GvR> test test_bsddb3 failed -- errors occurred; run in verbose
    GvR> mode for details

    GvR> Frankly, verbose mode waas too verbose to bother. :-(

None off the top of my head, but I'm swapping in my BerkeleyDB pages
to try to finish off support for 4.1.24.  I'll look into these when I
boot OSX again tonight.

-Barry


From gward@python.net  Mon Dec  2 17:06:09 2002
From: gward@python.net (Greg Ward)
Date: Mon, 2 Dec 2002 12:06:09 -0500
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <r01050400-1022-8708B8C205EA11D7ADDD003065D5E7E4@[10.0.0.23]>
References: <003001c298d7$550e8240$0201010a@wiktor> <r01050400-1022-8708B8C205EA11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <20021202170609.GA5209@cthulhu.gerg.ca>

On 02 December 2002, Just van Rossum said:
> I've been doing some thinking about the import mechanism, too, and my
> current gripe is about the __import__ hook: I find it utterly useless
> that the implementer of the hook is supposed to deal with
> sys.modules. What I wish for is a hook that only gets invoked if the
> module isn't already in sys.modules.

I can see where you're coming from, but one disadvantage to your scheme
is that it would not be possible to override .py/.pyc/.pyo files -- they
would always be imported in preference to any other scheme.  This could
well be a feature.  ;-)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
What happens if you touch these two wires tog--


From mwh@python.net  Mon Dec  2 17:38:54 2002
From: mwh@python.net (Michael Hudson)
Date: 02 Dec 2002 17:38:54 +0000
Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime obj_delta.c,1.14,1.15 test_both.py,1.15,1.16
In-Reply-To: tim_one@users.sourceforge.net's message of "Mon, 02 Dec 2002 09:31:39 -0800"
References: <E18IuPz-0003zz-00@sc8-pr-cvs1.sourceforge.net>
Message-ID: <2mlm38jp75.fsf@starship.python.net>

tim_one@users.sourceforge.net writes:

> + /* XXX A further broken attempt to get pickling to work.
> +  * XXX This avoids the problem above, but dies instead with
> +  * XXX    PicklingError: Can't pickle <type 'timedelta'>: it's not
> +  * XXX    found as __builtin__.tmedelta

You need to arrange for a module name to appear in timedelta's
tp_name:

Index: obj_delta.c
===================================================================
RCS file: /cvsroot/python/python/nondist/sandbox/datetime/obj_delta.c,v
retrieving revision 1.15
diff -c -r1.15 obj_delta.c
*** obj_delta.c 2 Dec 2002 17:31:21 -0000       1.15
--- obj_delta.c 2 Dec 2002 17:37:55 -0000
***************
*** 673,679 ****
  static PyTypeObject PyDateTime_DeltaType = {
        PyObject_HEAD_INIT(NULL)
        0,                                              /* ob_size */
!       "timedelta",                                    /* tp_name */
        sizeof(PyDateTime_Delta),                       /* tp_basicsize */
        0,                                              /* tp_itemsize */
        0,                                              /* tp_dealloc */
--- 673,679 ----
  static PyTypeObject PyDateTime_DeltaType = {
        PyObject_HEAD_INIT(NULL)
        0,                                              /* ob_size */
!       "datetime.timedelta",                           /* tp_name */
        sizeof(PyDateTime_Delta),                       /* tp_basicsize */
        0,                                              /* tp_itemsize */
        0,                                              /* tp_dealloc */

seems a good bet.

Cheers,
M.

-- 
  I'm not sure that the ability to create routing diagrams 
  similar to pretzels with mad cow disease is actually a 
  marketable skill.                                     -- Steve Levin
               -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html


From just@letterror.com  Mon Dec  2 17:50:32 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  2 Dec 2002 18:50:32 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <20021202170609.GA5209@cthulhu.gerg.ca>
Message-ID: <r01050400-1022-57CA743E062011D7ADDD003065D5E7E4@[10.0.0.23]>

Greg Ward wrote:

> On 02 December 2002, Just van Rossum said:
> > I've been doing some thinking about the import mechanism, too, and my
> > current gripe is about the __import__ hook: I find it utterly useless
> > that the implementer of the hook is supposed to deal with
> > sys.modules. What I wish for is a hook that only gets invoked if the
> > module isn't already in sys.modules.
> 
> I can see where you're coming from, but one disadvantage to your scheme
> is that it would not be possible to override .py/.pyc/.pyo files -- they
> would always be imported in preference to any other scheme.  This could
> well be a feature.  ;-)

I don't follow. In my scheme (which in the meantime I've partially implemented)
any first import will be done by an import hook on sys.import_hooks, so you have
full control. If you mean that you can't hook imports that are already in
sys.modules: yes, that's a feature ;-). On the other hand, the existing
__import__ hook will not change, so you can still do it using __import__. Look
at it this way: the default __import__ implements the new scheme; it's the low
level hook. For 99.9% of import hook needs you wouldn't touch it, though. Here's
the protocol that my current implementation uses:

    - sys.import_hooks is a list of hooks
    - a hook is a callable object taking two arguments:
      [sub]modulename and path. Path is either the parents package's
      __path__ or None.
    - a hook either returns None ("module not found") or a two-tuple:
      (loadfunc, cookie). loadfunc is a callable object that will
      be called with two arguments: the fullname and "cookie". The
      cookie is just an arbitrary object, private to the hook. The
      loaderfunc must return the imported module.
    - (Currently it expects that the loaderfunc will insert the module
      in sys.path, I'm not sure I like that. Actually, I am sure I
      don't like that <wink>, but changing that would require more
      changes to import.c that I'd like. I'll have to investigate.)

It plugs neatly into import_submodule(), right now it's a tiny patch.

Just


From tim.one@comcast.net  Mon Dec  2 18:35:58 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 02 Dec 2002 13:35:58 -0500
Subject: [Python-Dev] RE: how to kill process on Windows started with
 os.spawn?
In-Reply-To: <m3wums4vst.fsf@mira.informatik.hu-berlin.de>
Message-ID: <LNBBLJKPBEHFEDALKOLCCELDDAAB.tim.one@comcast.net>

[Tim]
>> Yes.  On Windows it wants a handle, because that's what spawn*()
>> returns on Windows.  But, again, Windows processes aren't intended
>> to be killed externally.

[martin@v.loewis.de]
> That sounds like FUD.

Argue w/ Microsoft -- you've read their docs.

> Why is it that processes aren't intended to be killed? Because the
> assassin can't know what state the process is in, so the process
> may not complete correctly?

It's easy to find long articles about the dangers on the net.  Here's one I
don't think has been posted before:

    http://tinyurl.com/35o6

> This is a general problem with killing, not specific to Windows, and
> it never stopped a killer.
>
> Or can you crash the operating system or the killer process by
> terminating some other process?

It's certainly possible to crash Win9x by killing processes, and easy to
hang the killer process (indeed, I have four shareware "process killers" on
my home box, as no single one of them is able to kill everything --
sometimes I have a hung process, and three hung process killers trying to
nuke it!  Win98SE generally becomes too unstable to continue running at that
point.).

>> Why do you need to kill a process externally?  For example, why
>> can't you make "please stop now" a part of the protocol, so that a
>> process can terminate itself gracefully when told to?

> Because the process did not respond to the protocol. Should I kill it
> anyway and send a bug report to Microsoft?

The questions were addressed to Skip about his specific app.  It sounded to
me like he wanted to kill() routinely, instead of pursuing a clean shutdown.

    The TerminateProcess function is used to unconditionally cause a
    process to exit.  Use it only in extreme circumstances.  The state
    of global data maintained by dynamic-link libraries (DLLs) may be
    compromised if TerminateProcess is used rather than ExitProcess.

because TerminateProcess doesn't notify attached DLLs of process
termination -- it's not solely the killed process's state that can get
hosed.



From tim.one@comcast.net  Mon Dec  2 18:42:35 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 02 Dec 2002 13:42:35 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime
 obj_delta.c,1.14,1.15 test_both.py,1.15,1.16
In-Reply-To: <2mlm38jp75.fsf@starship.python.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAELEDAAB.tim.one@comcast.net>

[Michael Hudson]
> You need to arrange for a module name to appear in timedelta's
> tp_name:

Ya, I figured that out.  The next surprise was the need to create a
__safe_for_unpickling__ attr in the type's dict.  This was confusing because
none of it was necessary for the Python implementation of this stuff, and
examples of the necessity for, and mechanics of, __reduce__, were hard to
find.



From gward@python.net  Mon Dec  2 19:05:59 2002
From: gward@python.net (Greg Ward)
Date: Mon, 2 Dec 2002 14:05:59 -0500
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <r01050400-1022-57CA743E062011D7ADDD003065D5E7E4@[10.0.0.23]>
References: <20021202170609.GA5209@cthulhu.gerg.ca> <r01050400-1022-57CA743E062011D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <20021202190559.GA5342@cthulhu.gerg.ca>

On 02 December 2002, Just van Rossum said:
> I don't follow. In my scheme (which in the meantime I've partially
> implemented) any first import will be done by an import hook on
> sys.import_hooks, so you have full control.

Ahh!  I get it, sorry.  Makes sense now.

> If you mean that you can't
> hook imports that are already in sys.modules: yes, that's a feature

Agreed.  Once something has been imported, the import hooks should not
be bothered with it again.

>     - sys.import_hooks is a list of hooks
>     - a hook is a callable object taking two arguments:
>       [sub]modulename and path. Path is either the parents package's
>       __path__ or None.
>     - a hook either returns None ("module not found") or a two-tuple:
>       (loadfunc, cookie). loadfunc is a callable object that will

Sounds sensible, but please s/cookie/data/ (or similar).

>       be called with two arguments: the fullname and "cookie". The
>       cookie is just an arbitrary object, private to the hook. The
>       loaderfunc must return the imported module.
>     - (Currently it expects that the loaderfunc will insert the module
>       in sys.path, I'm not sure I like that. Actually, I am sure I
               ^^^^

You really meant sys.module, right? 

>       don't like that <wink>, but changing that would require more
>       changes to import.c that I'd like. I'll have to investigate.)

I agree: if an import hook is not responsible for checking sys.modules
before doing an import, it should not be responsible for updating
sys.modules after an import.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Clarke's Law:
Any sufficiently advanced technology is indistinguishable from magic.


From just@letterror.com  Mon Dec  2 19:19:22 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  2 Dec 2002 20:19:22 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
Message-ID: <r01050400-1022-FA5BD96C062A11D7ADDD003065D5E7E4@[10.0.0.23]>

Greg Ward wrote:

> >     - a hook either returns None ("module not found") or a two-tuple:
> >       (loadfunc, cookie). loadfunc is a callable object that will
> 
> Sounds sensible, but please s/cookie/data/ (or similar).

Ok.

> >       be called with two arguments: the fullname and "cookie". The
> >       cookie is just an arbitrary object, private to the hook. The
> >       loaderfunc must return the imported module.
> >     - (Currently it expects that the loaderfunc will insert the module
> >       in sys.path, I'm not sure I like that. Actually, I am sure I
>                ^^^^
> 
> You really meant sys.module, right? 

sys.modules actually ;-)

> >       don't like that <wink>, but changing that would require more
> >       changes to import.c that I'd like. I'll have to investigate.)
> 
> I agree: if an import hook is not responsible for checking sys.modules
> before doing an import, it should not be responsible for updating
> sys.modules after an import.

I hadn't even looked at it that way, but yeah, that's an excellent rationale...
But: they _may_ do so if they desire; a module is even allowed to stuff a whole
different object into sys.modules instead of itself, so I'll have to be careful.

Just


From jeremy@alum.mit.edu  Mon Dec  2 21:41:21 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 2 Dec 2002 16:41:21 -0500
Subject: [Python-Dev] RE: [Python-checkins] python/nondist/sandbox/datetime
 obj_delta.c,1.14,1.15 test_both.py,1.15,1.16
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEMCDAAB.tim.one@comcast.net>
References: <15851.51138.218721.353223@grendel.zope.com>
 <LNBBLJKPBEHFEDALKOLCOEMCDAAB.tim.one@comcast.net>
Message-ID: <15851.54145.324810.106619@slothrop.zope.com>

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

  TP> I'd rather find a way to avoid __reduce__ entirely, though!

Me, too!

  TP> The Python implementation of these things didn't need it, and in
  TP> the date and datetime cases it's creating bigger pickles than it
  TP> should -- __getstate__ and __setstate__ already did all that was
  TP> necessary, and no more than that.  Supplying an argument tuple
  TP> for __reduce__'s benefit loses either way: I either put the real
  TP> date/datetime arguments there, but then the pickle is of a big
  TP> tuple rather than of a tiny string.  Or I put a dummy argument
  TP> tuple there and also include the tiny string for __setstate__,
  TP> but these constructors require at least 3 arguments so that the
  TP> "dummy argument tuple" consumes substantial space of its own.

  TP> So, as it stands, ignoring the new-style-class administrative
  TP> pickle bloat in the Python implementation, the *guts* of the
  TP> pickles produced by the Python implementation are substantially
  TP> smaller than those produced by the C implementation.

The __reduce__() approach seems to be (either or both) overly complex
and underdocumented.  It's a real shame that something simple like
__getstate__() and __setstate__() can't be made to work for new-style
classes.

If I recall correctly, the problem is that there is no way to
distinguish a user-defined new-style class from a builtin type from a
user-defined subclass of a builtin type.  As a result, there's no way
for pickle to decide if it should be looking for __getstate__() or
invoking the complicated machinery that allows subclasses of builtin
types to be pickleable.  Another victim of unification.

> I figured timedelta *was* "a class", although now I guess that, in
> this context, it's not "a class" but "a type".  That's what you get
> when you unify the concepts <wink>.

Jerem



From dave@boost-consulting.com  Mon Dec  2 22:33:25 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 02 Dec 2002 17:33:25 -0500
Subject: [Python-Dev] New universal import mechanism ( Re:
 [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <003001c298d7$550e8240$0201010a@wiktor> ("Wiktor Sadowski"'s
 message of "Sun, 1 Dec 2002 02:16:52 +0100")
References: <003001c298d7$550e8240$0201010a@wiktor>
Message-ID: <ubs44vyoa.fsf@boost-consulting.com>

"Wiktor Sadowski" <art@wiktorsadowski.com> writes:

> I think Python should have a more universal import mechanism , which could be done
> by adding a new C_CUSTOM type  to the definitions for dynamic loading (importdl.h) ,
> a new struct _customtab to import.h and a new function to Python API (import.c).
> Then any archive,database,internet import architecture could be implemented 
> in C extensions without messing with Python core.

As long as we're discussing changes to the import mechanism, I'll
mention a facility that some of my users want.  I apologize in advance
for not knowing whether the proposed patch covers this ability or not.

My users want extension modules to be able to control how their
attributes are imported when using "import *". In particular,
Boost.Python includes a kind of multimethod facility, and some of my
users would like it if

      from bpl_extension import *

caused multimethod definitions in bpl_extension to be /merged/ into
the current module, rather than replacing any multimethod bindings
with the same name in the current module.

-Dave

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From martin@v.loewis.de  Mon Dec  2 22:47:09 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 02 Dec 2002 23:47:09 +0100
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCELDDAAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCELDDAAB.tim.one@comcast.net>
Message-ID: <m3d6okyr6a.fsf@mira.informatik.hu-berlin.de>

Tim Peters <tim.one@comcast.net> writes:

> > That sounds like FUD.
> 
> Argue w/ Microsoft -- you've read their docs.

Apparently not the ones you've read. The documentation of
TerminateProcess does not include any warning, except for the
recommendation to only use it extreme circumstances, and that the
global state maintained by DLLs may be compromised.

Please correct me if I'm wrong, but on NT+, there isn't any global
state maintained by DLLs that could get corrupted. I interpret the
warning as suggesting that "normal" shutdown procedures should be used
if possible.

> It's easy to find long articles about the dangers on the net.  Here's one I
> don't think has been posted before:
> 
>     http://tinyurl.com/35o6

It mentions a number of methods to find the window, then explains that
sending WM_CLOSE may fail, to finally explain that TerminateProcess
should always work, and that you may lose data by using it.

This is exactly the same as on Unix: If you kill a process that was
just saving a file, the file may get corrupted. If there were unsaved
data, they are lost. This is no reason not to use TerminateProcess.

> It's certainly possible to crash Win9x by killing processes, and
> easy to hang the killer process (indeed, I have four shareware
> "process killers" on my home box, as no single one of them is able
> to kill everything -- sometimes I have a hung process, and three
> hung process killers trying to nuke it!

That sounds like a bug in the killers. TerminateProcess returns
immediately, according to the documentation. Now, the process may not
terminate if it is blocked in the kernel (the same issue exists on
Unix, leading to zombie processes). If the killer then invokes
WaitForProcessClose, it is not surprising if it hangs.

Your observations suggest "Don't use TerminateProcess on Win9x".

I would extend this warning to "Don't use Win9x" :-)

> because TerminateProcess doesn't notify attached DLLs of process
> termination -- it's not solely the killed process's state that can get
> hosed.

Can you explain what the "global state" of a DLL is? It may have
global state on Win9x, and be "attached" to a process - but I
understand there is nothing "global" on NT+; the code gets mapped into
the process (which gets cleaned up when the process is terminated),
and there are no shared data.

Regards,
Martin



From greg@cosc.canterbury.ac.nz  Mon Dec  2 22:58:36 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Tue, 03 Dec 2002 11:58:36 +1300 (NZDT)
Subject: [Python-Dev] RE: how to kill process on Windows started with os.spawn?
In-Reply-To: <m3wums4vst.fsf@mira.informatik.hu-berlin.de>
Message-ID: <200212022258.gB2Mwao28701@kuku.cosc.canterbury.ac.nz>

> Because the assassin can't know what state the process is in,
> so the process may not complete correctly?
> 
> This is a general problem with killing, not specific to Windows, and
> it never stopped a killer.

Psychotic Killer: [wielding knife and pulling back shower curtain]
                  HA! Prepare to die!

Female Programmer: No, wait! You can't kill me now, you don't
                  know what state I'm in!



From tim.one@comcast.net  Mon Dec  2 22:56:05 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 02 Dec 2002 17:56:05 -0500
Subject: [Python-Dev] RE: how to kill process on Windows started with
 os.spawn?
In-Reply-To: <m3d6okyr6a.fsf@mira.informatik.hu-berlin.de>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEMPDAAB.tim.one@comcast.net>

Martin, I'm not interested in arguing about this.  If you want to implement
os.kill() for Windows, go ahead.  I only ask that the docs repeat the
vendor's warnings against using it.  It's fine for adult use, where "adult"
means you know you're courting trouble and are prepared to accept the
consequences.



From greg@cosc.canterbury.ac.nz  Mon Dec  2 23:04:02 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Tue, 03 Dec 2002 12:04:02 +1300 (NZDT)
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: <200212021030.gB2AUfR24830@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <200212022304.gB2N42Z28704@kuku.cosc.canterbury.ac.nz>

Guido:

> Do you know if setting the stack limit actually allocates that much
> memory for the stack in the process in Mac OSX, or does it only
> reserve VM space (like on traditional Unix)?

Strictly speaking, in traditional Unix it doesn't do either,
it just sets a limit on how big the stack can grow. Actually
allocating stack pages is done on demand.

I expect this is what MacOSX does.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From mhammond@skippinet.com.au  Mon Dec  2 23:47:35 2002
From: mhammond@skippinet.com.au (Mark Hammond)
Date: Tue, 3 Dec 2002 10:47:35 +1100
Subject: [Python-Dev] ssl for Windows
Message-ID: <LCEPIIGDJPKCOIHOBJEPMEDLIAAA.mhammond@skippinet.com.au>

I would like to check in a patch for http://www.python.org/sf/576711, which
adds the _ssl module to Windows builds.

The only slightly controversial thing is how the build process operates.  I
have quoted the relevant part of PCbuild/readme.txt below.  test_socket_ssl
passes once this is built, and at least 2 other people have reported build
success.

Are there any objections to the scheme?  Should I check it in?

Mark.

_ssl
    Python wrapper for the secure sockets library.

    Get the latest source code for OpenSSL from
        http://www.openssl.org

    Unpack into the "dist" directory, retaining the folder name from
    the archive - for example, the latest stable OpenSSL will install as
        dist/openssl-0.9.6g

    You can (theoretically) use any version of OpenSSL you like - the
    build process will automatically select the latest version.

    You must also install ActivePerl from
        http://www.activestate.com/Products/ActivePerl/
    as this is used by the OpenSSL build process.  Complain to them <wink>

    The MSVC project simply invokes PCBuild/build_ssl.py to perform
    the build.  This Python script locates and builds your OpenSSL
    installation, then invokes a simple makefile to build the final .pyd.

    build_ssl.py attempts to catch the most common errors (such as not
    being able to find OpenSSL sources, or not being able to find a Perl
    that works with OpenSSL) and give a reasonable error message.
    If you have a problem that doesn't seem to be handled correctly
    (eg, you know you have ActivePerl but we can't find it), please take
    a peek at build_ssl.py and suggest patches.  Note that build_ssl.py
    should be able to be run directly from the command-line.

    build_ssl.py/MSVC isn't clever enough to clean OpenSSL - you must do
this
    by hand.





From just@letterror.com  Tue Dec  3 00:29:28 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 01:29:28 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-FA5BD96C062A11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <r01050400-1022-4B275522065611D7ADDD003065D5E7E4@[10.0.0.23]>

--41333375-49397760-3247867773=:9006
Content-Type: text/plain; Charset=US-ASCII
Content-Transfer-Encoding: 7bit

Folks,

I've attached a proof-of-concept patch to import.c that implements the
sys.import_hooks idea I outlined earlier. Here's an updated description.

  - sys.import_hooks is a list of hooks.
  - A hook is a callable object taking two arguments:
    'modulename' and 'path'. 'path' is either the parents package's
    __path__ or None, 'modulename' is *always* a "simple" name: no
    dots.
  - Hooks are never called for modules already in sys.modules.
  - A hook either returns None ("module not found") or a two-tuple:
    (loadfunc, data). loadfunc is a callable object that will
    be called with two arguments: the 'fullname' and 'data'. 'data'
    is just an arbitrary object, private to the hook. The loaderfunc
    must return the imported module. It _may_ add the module to
    sys.modules, but it doesn't need to.
  - A hook is not supposed to fail: it should return None or a
    (loader, data) tuple. If a hook _does_ fail with an exception,
    the import is aborted and subsequent hooks won't get called.
    The exception will be propagated to the caller. (XXX this is
    perhaps brittle; maybe errors need to be ignored by default,
    or a warning should be issued instead.)

sys.import_hooks currently gets filled with a default hook:
imp.builtin_import_hook. This is simply a wrapper around find_module() and
friends. Perhaps it's an idea to split it into discrete hooks for builtin,
frozen, and file system imports. This can be done later.

An alternative to the current default hook would be to keep sys.import_hooks
empty by default and fall back to find_module/load_module if no hook managed to
find/load the module. Pro: this would allow for some optimizations, as the hook
mechanism would be bypassed (and therefore less code in import.c). Con: it would
then be awkward to completely disable the builtin import mechanism, it's less
transparent, less elegant. (Ok, I thought a bit more about this: I don't like it
at all. All imports should be done by a hook on sys.import_hooks. If
sys.import_hooks is empty, _any_ new import will fail.)

Rationale:
- Make writing import hooks simpler; this new scheme avoids much of the
complexity involved in writing a traditional __import__ hook.

Real rationale:
- Allow the "Import from Zip archive" patch to be rewritten as an external
module, avoiding the huge mess it is right now, as it touches more or less all
of import.c and then some. Paul Moore is doing an admirable job to get it
working and cleaned up, but I honestly don't see this getting into the core:
it's such a huge and complex patch that it's almost impossible to properly
review. It's a maintainance nightmare waiting to happen. And let's face it,
import.c isn't all that maintainable as it is...

Longer term rationale:
- It would allow us to refactor the builtin import mechanism(s) by writing
specialized hooks, replacing the current monolithic find_module/load_module
code.

Con:
It is slower for first-time imports. If you care, go ahead and measure ;-)

Comments?

Just
--41333375-49397760-3247867773=:9006
Content-Type: application/octet-stream; Name="newimporthook.patch"; X-Mac-Type="54455854"; X-Mac-Creator="522A6368"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; Filename="newimporthook.patch"

SW5kZXg6IFB5dGhvbi9pbXBvcnQuYwo9PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09ClJDUyBm
aWxlOiAvY3Zzcm9vdC9weXRob24vcHl0aG9uL2Rpc3Qvc3JjL1B5dGhvbi9pbXBv
cnQuYyx2CnJldHJpZXZpbmcgcmV2aXNpb24gMi4yMTEKZGlmZiAtYyAtcjIuMjEx
IGltcG9ydC5jCioqKiBQeXRob24vaW1wb3J0LmMJMjkgTm92IDIwMDIgMjA6NDc6
NDAgLTAwMDAJMi4yMTEKLS0tIFB5dGhvbi9pbXBvcnQuYwkyIERlYyAyMDAyIDIz
OjQwOjUwIC0wMDAwCioqKioqKioqKioqKioqKgoqKiogMTA1LDExMCAqKioqCi0t
LSAxMDUsMTI4IC0tLS0KICB9OwogICNlbmRpZgogIAorIC8qIGltcG9ydF9ob29r
cyBzdHVmZiwgcHJvYmFibHkgbmVlZHMgcmVvcmdhbml6YXRpb24gKi8KKyAvKiBm
b3J3YXJkIGRlY2xzICovCisgc3RhdGljIEZJTEUgKmdldF9maWxlKGNoYXIgKnBh
dGhuYW1lLCBQeU9iamVjdCAqZm9iLCBjaGFyICptb2RlKTsKKyBzdGF0aWMgUHlP
YmplY3QgKmNhbGxfZmluZF9tb2R1bGUoY2hhciAqbmFtZSwgUHlPYmplY3QgKnBh
dGgpOworIAorIC8qICJtYW51YWwiIGNvbnN0cnVjdGlvbiBvZiB0d28gQ0Z1bmN0
aW9uIG9iamVjdHM7IGNhbid0IHVzZSB0aGUKKyAgICByZWd1bGFyIG1lY2hhbmlz
bSBhcyB3ZSBuZWVkIHRoZXNlIGZ1bmN0aW9uIG9iamVjdHMgZXZlbiBpZiB0aGUK
KyAgICBpbXAgbW9kdWxlIGlzbid0IGltcG9ydGVkLiAqLworIHN0YXRpYyBQeU9i
amVjdCAqaW1wX2J1aWx0aW5faW1wb3J0X2hvb2soUHlPYmplY3QgKnNlbGYsIFB5
T2JqZWN0ICphcmdzKTsKKyBzdGF0aWMgUHlNZXRob2REZWYgYnVpbHRpbl9pbXBv
cnRfaG9va19kZWYgPSAKKyAJeyJidWlsdGluX2ltcG9ydF9ob29rIiwgaW1wX2J1
aWx0aW5faW1wb3J0X2hvb2ssIE1FVEhfVkFSQVJHUywgIlhYWCJ9OworIHN0YXRp
YyBQeU9iamVjdCAqYnVpbHRpbl9pbXBvcnRfaG9va19mdW5jID0gTlVMTDsKKyAK
KyBzdGF0aWMgUHlPYmplY3QgKmltcF9idWlsdGluX2xvYWRlcihQeU9iamVjdCAq
c2VsZiwgUHlPYmplY3QgKmFyZ3MpOworIHN0YXRpYyBQeU1ldGhvZERlZiBidWls
dGluX2xvYWRlcl9kZWYgPSAKKyAJeyJidWlsdGluX2xvYWRlciIsIGltcF9idWls
dGluX2xvYWRlciwgTUVUSF9WQVJBUkdTLCAiWFhYIn07Cisgc3RhdGljIFB5T2Jq
ZWN0ICpidWlsdGluX2xvYWRlcl9mdW5jID0gTlVMTDsKKyAKICAvKiBJbml0aWFs
aXplIHRoaW5ncyAqLwogIAogIHZvaWQKKioqKioqKioqKioqKioqCioqKiAxMTQs
MTE5ICoqKioKLS0tIDEzMiwxMzggLS0tLQogIAlzdHJ1Y3QgZmlsZWRlc2NyICpm
aWxldGFiOwogIAlpbnQgY291bnREID0gMDsKICAJaW50IGNvdW50UyA9IDA7Cisg
CVB5T2JqZWN0ICppbXBvcnRfaG9va3M7CiAgCiAgCS8qIHByZXBhcmUgX1B5SW1w
b3J0X0ZpbGV0YWI6IGNvcHkgZW50cmllcyBmcm9tCiAgCSAgIF9QeUltcG9ydF9E
eW5Mb2FkRmlsZXRhYiBhbmQgX1B5SW1wb3J0X1N0YW5kYXJkRmlsZXRhYi4KKioq
KioqKioqKioqKioqCioqKiAxNTAsMTU1ICoqKioKLS0tIDE2OSwxODUgLS0tLQog
IAkJICAgY29kZSBjcmVhdGVkIGluIG5vcm1hbCBvcGVyYXRpb24gbW9kZS4gKi8K
ICAJCXB5Y19tYWdpYyA9IE1BR0lDICsgMTsKICAJfQorIAorIAkvKiBTZXR1cCBz
eXMuaW1wb3J0X2hvb2tzOyBjcmVhdGUgZnVuY3Rpb24gb2JqZWN0cyBmb3IgdGhl
CisgCSAgIGRlZmF1bHQgaG9vayBhbmQgbWF0Y2hpbmcgbG9hZGVyIGZ1bmN0aW9u
cy4gKi8KKyAJYnVpbHRpbl9pbXBvcnRfaG9va19mdW5jID0gUHlDRnVuY3Rpb25f
TmV3KCZidWlsdGluX2ltcG9ydF9ob29rX2RlZiwKKyAJCQkJCQkgICBOVUxMKTsK
KyAJYnVpbHRpbl9sb2FkZXJfZnVuYyA9IFB5Q0Z1bmN0aW9uX05ldygmYnVpbHRp
bl9sb2FkZXJfZGVmLCBOVUxMKTsKKyAJaW1wb3J0X2hvb2tzID0gUHlfQnVpbGRW
YWx1ZSgiW09dIiwgYnVpbHRpbl9pbXBvcnRfaG9va19mdW5jKTsKKyAJaWYgKGlt
cG9ydF9ob29rcyAhPSBOVUxMKSB7CisgCQlQeVN5c19TZXRPYmplY3QoImltcG9y
dF9ob29rcyIsIGltcG9ydF9ob29rcyk7CisgCQlQeV9ERUNSRUYoaW1wb3J0X2hv
b2tzKTsKKyAJfQogIH0KICAKICB2b2lkCioqKioqKioqKioqKioqKgoqKiogMjQ2
LDI1MiAqKioqCiAgCSJwYXRoIiwgImFyZ3YiLCAicHMxIiwgInBzMiIsICJleGl0
ZnVuYyIsCiAgCSJleGNfdHlwZSIsICJleGNfdmFsdWUiLCAiZXhjX3RyYWNlYmFj
ayIsCiAgCSJsYXN0X3R5cGUiLCAibGFzdF92YWx1ZSIsICJsYXN0X3RyYWNlYmFj
ayIsCiEgCU5VTEwKICB9OwogIAogIHN0YXRpYyBjaGFyKiBzeXNfZmlsZXNbXSA9
IHsKLS0tIDI3NiwyODIgLS0tLQogIAkicGF0aCIsICJhcmd2IiwgInBzMSIsICJw
czIiLCAiZXhpdGZ1bmMiLAogIAkiZXhjX3R5cGUiLCAiZXhjX3ZhbHVlIiwgImV4
Y190cmFjZWJhY2siLAogIAkibGFzdF90eXBlIiwgImxhc3RfdmFsdWUiLCAibGFz
dF90cmFjZWJhY2siLAohIAkiaW1wb3J0X2hvb2tzIiwgTlVMTAogIH07CiAgCiAg
c3RhdGljIGNoYXIqIHN5c19maWxlc1tdID0gewoqKioqKioqKioqKioqKioKKioq
IDE5NjEsMTk2NiAqKioqCi0tLSAxOTkxLDIxMTAgLS0tLQogIH0KICAKICBzdGF0
aWMgUHlPYmplY3QgKgorIGltcF9idWlsdGluX2ltcG9ydF9ob29rKFB5T2JqZWN0
ICpzZWxmLCBQeU9iamVjdCAqYXJncykgeworIAljaGFyICpuYW1lOworIAlQeU9i
amVjdCAqcGF0aCwgKnJlczsKKyAJCisgCWlmICghUHlBcmdfUGFyc2VUdXBsZShh
cmdzLCAic08iLCAmbmFtZSwgJnBhdGgpKQorIAkJcmV0dXJuIE5VTEw7CisgCWlm
IChwYXRoID09IFB5X05vbmUpCisgCQlwYXRoID0gTlVMTDsKKyAJcmVzID0gY2Fs
bF9maW5kX21vZHVsZShuYW1lLCBwYXRoKTsKKyAJaWYgKHJlcyA9PSBOVUxMKSB7
CisgCQlpZiAoIVB5RXJyX0V4Y2VwdGlvbk1hdGNoZXMoUHlFeGNfSW1wb3J0RXJy
b3IpKQorIAkJCXJldHVybiBOVUxMOworIAkJUHlFcnJfQ2xlYXIoKTsKKyAJCVB5
X0lOQ1JFRihQeV9Ob25lKTsKKyAJCXJldHVybiBQeV9Ob25lOworIAl9CisgCXJl
dHVybiBQeV9CdWlsZFZhbHVlKCIoT08pIiwgYnVpbHRpbl9sb2FkZXJfZnVuYywg
cmVzKTsKKyB9CisgCisgc3RhdGljIFB5T2JqZWN0ICoKKyBpbXBfYnVpbHRpbl9s
b2FkZXIoUHlPYmplY3QgKnNlbGYsIFB5T2JqZWN0ICphcmdzKSB7CisgCWNoYXIg
KmZ1bGxuYW1lLCAqcGF0aG5hbWUsICpzdWZmaXgsICptb2RlOworIAlpbnQgdHlw
ZTsKKyAJUHlPYmplY3QgKmRhdGEsICpmb2IsICptb2Q7CisgCUZJTEUgKmZwOwor
IAorIAlpZiAoIVB5QXJnX1BhcnNlVHVwbGUoYXJncywgInNPOmJ1aWx0aW5fbG9h
ZGVyIiwKKyAJCQkgICAgICAmZnVsbG5hbWUsICZkYXRhKSkKKyAJCXJldHVybiBO
VUxMOworIAlpZiAoIVB5QXJnX1BhcnNlVHVwbGUoZGF0YSwgIk9zKHNzaSk6YnVp
bHRpbl9sb2FkZXIiLAorIAkJCSAgICAgICZmb2IsICZwYXRobmFtZSwgJnN1ZmZp
eCwKKyAJCQkgICAgICAmbW9kZSwgJnR5cGUpKQorIAkJcmV0dXJuIE5VTEw7Cisg
CS8qIFhYWCBjaGVjayBtb2RlLCBhcyBpbiBsb2FkX21vZHVsZSwgYnV0IHRoZSBj
b2RlIG5lZWRzIHJlZmFjdG9yaW5nICovCisgCWlmIChmb2IgPT0gUHlfTm9uZSkK
KyAJCWZwID0gTlVMTDsKKyAJZWxzZSB7CisgCQlpZiAoIVB5RmlsZV9DaGVjayhm
b2IpKSB7CisgCQkJUHlFcnJfU2V0U3RyaW5nKFB5RXhjX1ZhbHVlRXJyb3IsCisg
CQkJCSJsb2FkX21vZHVsZSBhcmcjMiBzaG91bGQgYmUgYSBmaWxlIG9yIE5vbmUi
KTsKKyAJCQlyZXR1cm4gTlVMTDsKKyAJCX0KKyAJCWZwID0gZ2V0X2ZpbGUocGF0
aG5hbWUsIGZvYiwgbW9kZSk7CisgCQlpZiAoZnAgPT0gTlVMTCkKKyAJCQlyZXR1
cm4gTlVMTDsKKyAJfQorIAltb2QgPSBsb2FkX21vZHVsZShmdWxsbmFtZSwgZnAs
IHBhdGhuYW1lLCB0eXBlKTsKKyAJaWYgKGZwKQorIAkJZmNsb3NlKGZwKTsKKyAJ
cmV0dXJuIG1vZDsKKyB9CisgCisgc3RhdGljIFB5T2JqZWN0ICoKKyBmaW5kX21v
ZHVsZTIoY2hhciAqc3VibmFtZSwgUHlPYmplY3QgKnBhdGgpCisgeworIAlpbnQg
aSwgbmhvb2tzOworIAlQeU9iamVjdCAqYXJncywgKnJlcyA9IE5VTEw7CisgCVB5
T2JqZWN0ICppbXBvcnRfaG9va3MgPSBQeVN5c19HZXRPYmplY3QoImltcG9ydF9o
b29rcyIpOworIAorIAlpZiAoaW1wb3J0X2hvb2tzID09IE5VTEwgfHwgIVB5TGlz
dF9DaGVjayhpbXBvcnRfaG9va3MpKSB7CisgCQlQeUVycl9TZXRTdHJpbmcoUHlF
eGNfSW1wb3J0RXJyb3IsCisgCQkJCSJzeXMuaW1wb3J0X2hvb2tzIG11c3QgYmUg
YSBsaXN0IG9mIGltcG9ydCBob29rcyIpOworIAkJcmV0dXJuIE5VTEw7CisgCX0K
KyAJaWYgKHBhdGggPT0gTlVMTCkKKyAJCXBhdGggPSBQeV9Ob25lOworIAlhcmdz
ID0gUHlfQnVpbGRWYWx1ZSgiKHNPKSIsIHN1Ym5hbWUsIHBhdGgpOworIAlpZiAo
YXJncyA9PSBOVUxMKQorIAkJcmV0dXJuIE5VTEw7CisgCW5ob29rcyA9IFB5TGlz
dF9TaXplKGltcG9ydF9ob29rcyk7CisgCWZvciAoaSA9IDA7IGkgPCBuaG9va3M7
IGkrKykgeworIAkJUHlPYmplY3QgKmhvb2sgPSBQeUxpc3RfR2V0SXRlbShpbXBv
cnRfaG9va3MsIGkpOworIAkJaWYgKGhvb2sgPT0gTlVMTCkKKyAJCQlnb3RvIGVy
cm9yOworIAkJcmVzID0gUHlPYmplY3RfQ2FsbChob29rLCBhcmdzLCBOVUxMKTsK
KyAJCWlmIChyZXMgPT0gTlVMTCkKKyAJCQlnb3RvIGVycm9yOworIAkJaWYgKHJl
cyAhPSBQeV9Ob25lKQorIAkJCWJyZWFrOworIAkJUHlfREVDUkVGKHJlcyk7Cisg
CQlyZXMgPSBOVUxMOworIAkJY29udGludWU7CisgCX0KKyAJaWYgKHJlcyA9PSBO
VUxMKSB7CisgCQlQeV9JTkNSRUYoUHlfTm9uZSk7CisgCQlyZXMgPSBQeV9Ob25l
OworIAl9CisgZXJyb3I6CisgCVB5X0RFQ1JFRihhcmdzKTsKKyAJcmV0dXJuIHJl
czsKKyB9CisgCisgc3RhdGljIFB5T2JqZWN0ICoKKyBsb2FkX21vZHVsZTIoY2hh
ciAqZnVsbG5hbWUsIFB5T2JqZWN0ICpmaW5kZXJfcmVzdWx0KQorIHsKKyAJUHlP
YmplY3QgKmxvYWRlcmZ1bmMsICpkYXRhLCAqYXJncywgKm1vZDsKKyAJCisgCWlm
ICghUHlBcmdfUGFyc2VUdXBsZShmaW5kZXJfcmVzdWx0LCAiT08iLCAmbG9hZGVy
ZnVuYywgJmRhdGEpKQorIAkJcmV0dXJuIE5VTEw7CisgCWFyZ3MgPSBQeV9CdWls
ZFZhbHVlKCIoc08pIiwgZnVsbG5hbWUsIGRhdGEpOworIAlpZiAoYXJncyA9PSBO
VUxMKQorIAkJcmV0dXJuIE5VTEw7CisgCW1vZCA9IFB5T2JqZWN0X0NhbGwobG9h
ZGVyZnVuYywgYXJncywgTlVMTCk7CisgCVB5X0RFQ1JFRihhcmdzKTsKKyAJaWYg
KG1vZCAhPSBOVUxMICYmIG1vZCAhPSBQeV9Ob25lKSB7CisgCQkvKiBpbnNlcnQg
bW9kdWxlIGluIHN5cy5tb2R1bGVzLCBpZiBpdCBpc24ndCB0aGVyZSB5ZXQgKi8K
KyAJCVB5T2JqZWN0ICptb2R1bGVzID0gUHlJbXBvcnRfR2V0TW9kdWxlRGljdCgp
OworIAkJaWYgKFB5RGljdF9HZXRJdGVtU3RyaW5nKG1vZHVsZXMsIGZ1bGxuYW1l
KSA9PSBOVUxMKQorIAkJCVB5RGljdF9TZXRJdGVtU3RyaW5nKG1vZHVsZXMsIGZ1
bGxuYW1lLCBtb2QpOworIAl9CisgCXJldHVybiBtb2Q7CisgfQorIAorIHN0YXRp
YyBQeU9iamVjdCAqCiAgaW1wb3J0X3N1Ym1vZHVsZShQeU9iamVjdCAqbW9kLCBj
aGFyICpzdWJuYW1lLCBjaGFyICpmdWxsbmFtZSkKICB7CiAgCVB5T2JqZWN0ICpt
b2R1bGVzID0gUHlJbXBvcnRfR2V0TW9kdWxlRGljdCgpOwoqKioqKioqKioqKioq
KioKKioqIDE5NzYsMTk4NCAqKioqCiAgCX0KICAJZWxzZSB7CiAgCQlQeU9iamVj
dCAqcGF0aDsKLSAJCWNoYXIgYnVmW01BWFBBVEhMRU4rMV07Ci0gCQlzdHJ1Y3Qg
ZmlsZWRlc2NyICpmZHA7Ci0gCQlGSUxFICpmcCA9IE5VTEw7CiAgCiAgCQlpZiAo
bW9kID09IFB5X05vbmUpCiAgCQkJcGF0aCA9IE5VTEw7Ci0tLSAyMTIwLDIxMjUg
LS0tLQoqKioqKioqKioqKioqKioKKioqIDE5OTEsMjAwOSAqKioqCiAgCQkJfQog
IAkJfQogIAohIAkJYnVmWzBdID0gJ1wwJzsKISAJCWZkcCA9IGZpbmRfbW9kdWxl
KHN1Ym5hbWUsIHBhdGgsIGJ1ZiwgTUFYUEFUSExFTisxLCAmZnApOwohIAkJUHlf
WERFQ1JFRihwYXRoKTsKISAJCWlmIChmZHAgPT0gTlVMTCkgewohIAkJCWlmICgh
UHlFcnJfRXhjZXB0aW9uTWF0Y2hlcyhQeUV4Y19JbXBvcnRFcnJvcikpCiEgCQkJ
CXJldHVybiBOVUxMOwohIAkJCVB5RXJyX0NsZWFyKCk7CiAgCQkJUHlfSU5DUkVG
KFB5X05vbmUpOwogIAkJCXJldHVybiBQeV9Ob25lOwogIAkJfQohIAkJbSA9IGxv
YWRfbW9kdWxlKGZ1bGxuYW1lLCBmcCwgYnVmLCBmZHAtPnR5cGUpOwohIAkJaWYg
KGZwKQohIAkJCWZjbG9zZShmcCk7CiAgCQlpZiAobW9kICE9IFB5X05vbmUpIHsK
ICAJCQkvKiBJcnJlc3BlY3RpdmUgb2YgdGhlIHN1Y2Nlc3Mgb2YgdGhpcyBsb2Fk
LCBtYWtlIGEKICAJCQkgICByZWZlcmVuY2UgdG8gaXQgaW4gdGhlIHBhcmVudCBw
YWNrYWdlIG1vZHVsZS4KLS0tIDIxMzIsMjE0OCAtLS0tCiAgCQkJfQogIAkJfQog
IAohIAkJcmVzID0gZmluZF9tb2R1bGUyKHN1Ym5hbWUsIHBhdGgpOwohIAkJaWYg
KHJlcyA9PSBOVUxMKQohIAkJCS8qIGZpbmRpbmcgcmFpc2VkIGFuIGVycm9yICov
CiEgCQkJcmV0dXJuIE5VTEw7CiEgCQlpZiAocmVzID09IFB5X05vbmUpIHsKISAJ
CQkvKiB0aGUgbW9kdWxlIHdhc24ndCBmb3VuZCAqLwogIAkJCVB5X0lOQ1JFRihQ
eV9Ob25lKTsKICAJCQlyZXR1cm4gUHlfTm9uZTsKICAJCX0KISAJCW0gPSBsb2Fk
X21vZHVsZTIoZnVsbG5hbWUsIHJlcyk7CiEgCiAgCQlpZiAobW9kICE9IFB5X05v
bmUpIHsKICAJCQkvKiBJcnJlc3BlY3RpdmUgb2YgdGhlIHN1Y2Nlc3Mgb2YgdGhp
cyBsb2FkLCBtYWtlIGEKICAJCQkgICByZWZlcmVuY2UgdG8gaXQgaW4gdGhlIHBh
cmVudCBwYWNrYWdlIG1vZHVsZS4KKioqKioqKioqKioqKioqCioqKiAyNzI0LDI3
MjkgKioqKgotLS0gMjg2MywyODcyIC0tLS0KICAJaWYgKHNldGludChkLCAiUFlf
RlJPWkVOIiwgUFlfRlJPWkVOKSA8IDApIGdvdG8gZmFpbHVyZTsKICAJaWYgKHNl
dGludChkLCAiUFlfQ09ERVJFU09VUkNFIiwgUFlfQ09ERVJFU09VUkNFKSA8IDAp
IGdvdG8gZmFpbHVyZTsKICAKKyAJaWYgKFB5RGljdF9TZXRJdGVtU3RyaW5nKGQs
ICJidWlsdGluX2ltcG9ydF9ob29rIiwgYnVpbHRpbl9pbXBvcnRfaG9va19mdW5j
KSA8IDApCisgCQlnb3RvIGZhaWx1cmU7CisgCWlmIChQeURpY3RfU2V0SXRlbVN0
cmluZyhkLCAiYnVpbHRpbl9sb2FkZXIiLCBidWlsdGluX2xvYWRlcl9mdW5jKSA8
IDApCisgCQlnb3RvIGZhaWx1cmU7CiAgICBmYWlsdXJlOgogIAk7CiAgfQo=
--41333375-49397760-3247867773=:9006--


From skip@pobox.com  Mon Dec  2 18:43:01 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 2 Dec 2002 12:43:01 -0600
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net>
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net>
 <15850.58353.356475.182137@gargle.gargle.HOWL>
 <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15851.43445.443010.182362@montanaro.dyndns.org>

    Guido> I'm still seeing errors in test_socket; today (on a different
    Guido> network) I got this:

    Guido> test test_socket failed -- Traceback (most recent call last):
    Guido>   File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes
    Guido>     ip = socket.gethostbyname(hostname)
    Guido> gaierror: (7, 'No address associated with nodename')

What does the hostname(1) command return?  By default I think it generates
some oddball thing like "Montanaro.local.".  Apple calls it a Rendevous
name.  (I've no idea what that is.)  The fix for me was recommended by
another group: get a dyndns.org account and change the HOSTNAME line in
/etc/hostconfig from

    HOSTNAME=-AUTOMATIC-

to

    HOSTNAME=montanaro.dyndns.org

Skip


From skip@pobox.com  Mon Dec  2 18:36:47 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 2 Dec 2002 12:36:47 -0600
Subject: [Python-Dev] RE: how to kill process on Windows started with
 os.spawn?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIMDAAB.tim.one@comcast.net>
References: <15850.37138.572612.258716@montanaro.dyndns.org>
 <LNBBLJKPBEHFEDALKOLCEEIMDAAB.tim.one@comcast.net>
Message-ID: <15851.43071.25770.622477@montanaro.dyndns.org>

    >> In my environment I need to tunnel SB's pop3proxy app through an ssh
    >> connection.  

    Tim> Why do you need to kill a process externally?  For example, why
    Tim> can't you make "please stop now" a part of the protocol, so that a
    Tim> process can terminate itself gracefully when told to?

I start the background ssh process from within pop3proxy, so it seems to me
I ought to be the one to terminate it.  As for protocol, I'm not aware of
another way to terminate a running ssh other than to call kill(2) with a
specified signal.  That's more-or-less the Unix way.  I don't have the
luxury of modifying ssh to understand some application-specific protocol.

Skip


From bkline@rksystems.com  Tue Dec  3 02:47:12 2002
From: bkline@rksystems.com (Bob Kline)
Date: Mon, 2 Dec 2002 21:47:12 -0500 (EST)
Subject: [Python-Dev] Getting python-bz2 into 2.3
In-Reply-To: <200211301531.gAUFV4T12368@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <Pine.LNX.4.33.0212021954200.8299-100000@rksystems.com>

On Sat, 30 Nov 2002, Guido van Rossum wrote:

> More testing is always welcome; if you could test-drive the
> instructions in PCbuild/readme.txt for Tcl/Tk and Sleepycat Berkeley
> DB on multiple platforms that would be super.

The instructions for building Tcl and Tk worked as written with no
problems (though I would have no idea where to begin with the XXX
questions, such as "Should we compile with OPTS=threads?").  The project
file for the _tkinter module also worked without any hitches, but I
don't see a test suite (no surprise for a GUI-based module) and I have
no clue about how to test it or even get some of my simple _tkinter
program to run.  I'm so far out of my depth here with respect to Tcl/Tk
that I won't waste list bandwidth to pursue this any further.

As for the Berkeley DB package, that too built without any difficulties.
The instructions conclude:

==============================  snip  ==============================
    XXX We're actually linking against Release_static\libdb40s.lib.
    XXX This yields the following warnings:
"""
Compiling...
_bsddb.c
Linking...
   Creating library ./_bsddb.lib and object ./_bsddb.exp
LINK : warning LNK4049: locally defined symbol "_malloc" imported
LINK : warning LNK4049: locally defined symbol "_free" imported
LINK : warning LNK4049: locally defined symbol "_fclose" imported
LINK : warning LNK4049: locally defined symbol "_fopen" imported
_bsddb.pyd - 0 error(s), 4 warning(s)
"""
    XXX This isn't encouraging, but I don't know what to do about it.
==============================  snip  ==============================

I did a little poking around, and found that the problem is that the
static library against which the _bsddb package is built is compiled
with the -MT option, which causes the multi-threaded runtime libraries
to be statically linked into libdb40s.lib, hence the warnings.  The
warnings should probably be taken seriously, because I think they mean
that we would end up with (for example) two sets of static heap control
information.

The instructions in PCBuild/readme.txt were actually a little confusing,
because right before the part I just quoted above tells us to "[c]hoose
configuration "db_buildall - Win32 Release", and build db_buildall.exe."
That version uses the -MD compilation option, which uses the DLL version
of the runtime libraries (which is what we want), but it creates a DLL
for libdb40 instead of a library which can be statically linked into
_bsddb.pyd.  So it looks as there are two ways to eliminate the warnings
above:

 1. Use the DLL version of libdb40 and link to its import library.
 2. Create a third configuration for the underlying Berkeley DB
    library, using the -MD compilation option, but building a
    statically linkable library instead of a DLL.

I would have assumed that the second option was needed, because I
assumed that Python wouldn't want to drag around lots of DLLs for the
various packages that get installed, but then I noticed that the
_tkinter package is doing just that, so I guess I'm not sure.

If the second option is used, it should presumably be done with the
cooperation of the folks behind the Berkeley DB library, so it wouldn't
be necessary to recreate the third configuration by hand each time a new
release of the library came out.

Finally, I was reluctant to dig any further into this package because I
read at the top of the instructions:

    "XXX The Sleepycat release we use will probably change before
     XXX 2.3a1."

and I didn't know whether that referred to a version upgrade or a
complete replacement of the underlying DB engine with some other
library, which would make any further work throwaway.

-- 
Bob Kline
mailto:bkline@rksystems.com
http://www.rksystems.com




From bac@OCF.Berkeley.EDU  Tue Dec  3 03:51:11 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Mon, 2 Dec 2002 19:51:11 -0800 (PST)
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: <15851.43445.443010.182362@montanaro.dyndns.org>
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net>
 <15850.58353.356475.182137@gargle.gargle.HOWL>
 <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net>
 <15851.43445.443010.182362@montanaro.dyndns.org>
Message-ID: <Pine.SOL.4.50.0212021949400.13161-100000@death.OCF.Berkeley.EDU>

[Skip Montanaro]

> What does the hostname(1) command return?  By default I think it generates
> some oddball thing like "Montanaro.local.".  Apple calls it a Rendevous
> name.  (I've no idea what that is.)  The fix for me was recommended by

Rendevous is Apple's implementation of IEEE's zero-conf.  The "oddball"
name is what is used  for the local DNS entries used by Rendevous since it
stores info by name instead of IP.

-Brett


From tim.one@comcast.net  Tue Dec  3 04:17:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 02 Dec 2002 23:17:30 -0500
Subject: [Python-Dev] ssl for Windows
In-Reply-To: <LCEPIIGDJPKCOIHOBJEPMEDLIAAA.mhammond@skippinet.com.au>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEODDAAB.tim.one@comcast.net>

[Mark Hammond]
> I would like to check in a patch for
> http://www.python.org/sf/576711, which
> adds the _ssl module to Windows builds.

Good work, Mark.  Don't wait for me -- it appears that most of my Windows
Python work has become "spare time" activity.

> The only slightly controversial thing is how the build process
> operates.

It's just a bit of a mess.  I don't mind installing a current Perl, if
that's what it takes.

> I have quoted the relevant part of PCbuild/readme.txt below.
> test_socket_ssl passes once this is built, and at least 2 other
> people have reported build success.
>
> Are there any objections to the scheme?  Should I check it in?

Please do; that will force the issues in a helpful way.



From skip@pobox.com  Tue Dec  3 04:29:09 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 2 Dec 2002 22:29:09 -0600
Subject: [Python-Dev] extension module .o files wind up in the wrong place
Message-ID: <15852.13077.577684.9372@montanaro.dyndns.org>

I notice that when building Python in a subdirectory of the src dir that the
.o files for extension modules are written in the wrong directory.
Graphically, if my source tree looks like

    head/dist/src
        Makefile.pre.in
        /Modules
        /Python
        ...
        build
            Makefile
            /Modules
            /Python
            ...

If I run 

    cd build
    ../configure --prefix=/Users/skip/local
    make

extension module .o files wind up in .../src/Modules instead of
.../src/build/Modules.

I don't have time to look into this right now.  I only just noticed it while
I was trying to do parallel builds in build.dbg and build.opt directories.
The debug .o files got written first, so when the optimized make came along,
there were already .o files, which had been compiled with --with-pydebug,
got linked improperly.

Skip



From greg@cosc.canterbury.ac.nz  Tue Dec  3 04:48:55 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Tue, 03 Dec 2002 17:48:55 +1300 (NZDT)
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: <15852.13077.577684.9372@montanaro.dyndns.org>
Message-ID: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz>

Skip Montanaro <skip@pobox.com>:

> extension module .o files wind up in .../src/Modules instead of
> .../src/build/Modules.

If this involves VPATH, I think I might know what's happening --
something similar happened to me recently.

Have you by any chance previously done a non-VPATH build in the source
tree?

It seems that, when using a VPATH, any pathname referencing something
in the build directory for which a corresponding file exists in the
VPATH directory gets changed to refer to the VPATH directory. So you
need to make sure there are *only* source files in the source tree
before doing a build using a VPATH.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From skip@pobox.com  Tue Dec  3 05:02:12 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 2 Dec 2002 23:02:12 -0600
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz>
References: <15852.13077.577684.9372@montanaro.dyndns.org>
 <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz>
Message-ID: <15852.15060.656223.659613@montanaro.dyndns.org>

    Greg> So you need to make sure there are *only* source files in the
    Greg> source tree before doing a build using a VPATH.

Yup, "cvs up" followed by "make distclean", then

    mkdir build
    cd build
    ../configure --prefix=/Users/skip/local
    make

Skip


From Wiktor Sadowski" <art@wiktorsadowski.com  Mon Dec  2 05:22:43 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Mon, 2 Dec 2002 06:22:43 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
Message-ID: <069901c299c2$d7988f80$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_0696_01C299CB.38DFD840
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

Just van Rossum  wrote:

>>Could you please post a normal diff of your patch? It looks very =
interesting.
Sorry, I haven't got time for cleaning up my modified Python C code and  =
building the diffs.

>.I'm not entirely sure yet how my idea and yours could be integrated, =
but I think
> it's work a look.

The concept is the same , there are differences in implementations.
The implementation you have posted is much more pure Python oriented.
I am using C structures and pointers to C functions (wherever possible) =
instead of Python objects=20
(well PyObjects are still C structures) , so maybe it's a little faster =
(maybe not)=20

Looks like I was wasting my time.;)

Wiktor Sadowski
art@wiktorsadowski.com





------=_NextPart_000_0696_01C299CB.38DFD840
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2><FONT face=3D"Times New Roman" =
size=3D3>Just van=20
Rossum&nbsp; wrote:</FONT><BR></FONT></DIV>
<DIV>&gt;&gt;Could you please post a normal diff of your patch? It looks =
very=20
interesting.<BR>Sorry, I haven't got time for cleaning up my modified =
Python C=20
code and&nbsp;&nbsp;building the diffs.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2><FONT face=3D"Times New Roman" =
size=3D3>&gt;.I'm not=20
entirely sure yet how my idea and yours could be integrated, but I =
think<BR>&gt;=20
it's work a look.</FONT><BR></FONT></DIV>
<DIV>The concept is the same , there are differences in =
implementations.</DIV>
<DIV>The&nbsp;implementation you have posted is much more pure Python=20
oriented.</DIV>
<DIV>I am using C structures and pointers to C functions (wherever =
possible)=20
instead of Python objects </DIV>
<DIV>(well&nbsp;PyObjects are still C structures) , so&nbsp;maybe =
it's&nbsp;a=20
little faster (maybe not) </DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Looks like I was wasting my time.;)</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Wiktor Sadowski</DIV>
<DIV><FONT face=3DArial size=3D2><A=20
href=3D"mailto:art@wiktorsadowski.com">art@wiktorsadowski.com</A></FONT><=
/DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>&nbsp;</DIV>
<DIV>&nbsp;</DIV></FONT></BODY></HTML>

------=_NextPart_000_0696_01C299CB.38DFD840--



From tim.one@comcast.net  Tue Dec  3 05:02:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 03 Dec 2002 00:02:06 -0500
Subject: [Python-Dev] Getting python-bz2 into 2.3
In-Reply-To: <Pine.LNX.4.33.0212021954200.8299-100000@rksystems.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEOGDAAB.tim.one@comcast.net>

[Bob Kline, tries building a more-or-less complete Windows Python]
> The instructions for building Tcl and Tk worked as written with no
> problems (though I would have no idea where to begin with the XXX
> questions, such as "Should we compile with OPTS=threads?").

That's OK, neither do I <wink>.  We didn't before (8.3.2), and changing
anything is risky.

> The project file for the _tkinter module also worked without any
> hitches, but I don't see a test suite (no surprise for a GUI-based
> module) and I have no clue about how to test it or even get some of
> my simple _tkinter program to run.  I'm so far out of my depth here
> with respect to Tcl/Tk that I won't waste list bandwidth to pursue
> this any further.

I don't know how to test it either.  I install Python after building it,
then try Tkinter._test() from a DOS-box Python shell, and try a bunch of
stuff in IDLE.  That all looked fine.  It was more worrisome to me that some
of Tcl and Tk's own tests didn't pass, but I can't dig into that.

> As for the Berkeley DB package, that too built without any difficulties.
> The instructions conclude:
>
> ==============================  snip  ==============================
>     XXX We're actually linking against Release_static\libdb40s.lib.
>     XXX This yields the following warnings:
> """
> Compiling...
> _bsddb.c
> Linking...
>    Creating library ./_bsddb.lib and object ./_bsddb.exp
> LINK : warning LNK4049: locally defined symbol "_malloc" imported
> LINK : warning LNK4049: locally defined symbol "_free" imported
> LINK : warning LNK4049: locally defined symbol "_fclose" imported
> LINK : warning LNK4049: locally defined symbol "_fopen" imported
> _bsddb.pyd - 0 error(s), 4 warning(s)
> """
>     XXX This isn't encouraging, but I don't know what to do about it.
> ==============================  snip  ==============================
>
> I did a little poking around, and found that the problem is that the
> static library against which the _bsddb package is built is compiled
> with the -MT option, which causes the multi-threaded runtime libraries
> to be statically linked into libdb40s.lib, hence the warnings.  The
> warnings should probably be taken seriously, because I think they mean
> that we would end up with (for example) two sets of static heap control
> information.

Right, but it's not clear that it matters.  For example, it's unlikely
there's a case where memory is allocated by something in libdb40s but freed
outside it, or vice versa.  Then two heaps are just two distinct heaps.  I'm
inclined to ignore it for now, since we're probably going to move to a
current Sleepycat release anyway (but I don't know more about that -- Barry
is straightening that out).

It's quite unclear to me what good libdb40s.lib *is* if anyone using, e.g.,
malloc, can't link it in safely!

> The instructions in PCBuild/readme.txt were actually a little
> confusing, because right before the part I just quoted above tells us
> to "[c]hoose configuration "db_buildall - Win32 Release", and build
> db_buildall.exe."

It just echoes the Sleepycat instructions here.

> That version uses the -MD compilation option,

In part.  It also triggers the building of subprojects that end up creating
libdb40s.lib too.

> which uses the DLL version of the runtime libraries (which is what we
> want), but it creates a DLL for libdb40 instead of a library which can
> be statically linked into _bsddb.pyd.

It creates both the DLL and the static library.

> So it looks as there are two ways to eliminate the warnings
> above:
>
>  1. Use the DLL version of libdb40 and link to its import library.

This works but is clumsier:  you can't run the Python tests then from the
build directory, without first physically copying the Sleepycat DLL into
PCbuild, and you *also* need to build-and-copy a distinct debug-mode
Sleepycat DLL else you can't run the debug-build Python tests.  In addition,
the installer needs to be fiddled then to copy more files.  Like the old
1.85 bsddb Windows port from Sam Rushing, and like the current bz2 and zlib
modules, life is easier all around if 3rd-party code can be sucked directly
into the Python wrapper .pyd file.

>  2. Create a third configuration for the underlying Berkeley DB
>     library, using the -MD compilation option, but building a
>     statically linkable library instead of a DLL.

I'm not sure how to do that, but don't want to muck with Sleepycat's build
process regardless.

> I would have assumed that the second option was needed, because I
> assumed that Python wouldn't want to drag around lots of DLLs for the
> various packages that get installed, but then I noticed that the
> _tkinter package is doing just that, so I guess I'm not sure.

It doesn't want to, but Tcl/Tk are big and do their own kinds of linking
tricks.

> If the second option is used, it should presumably be done with the
> cooperation of the folks behind the Berkeley DB library, so it wouldn't
> be necessary to recreate the third configuration by hand each time a
> new release of the library came out.
>
> Finally, I was reluctant to dig any further into this package because I
> read at the top of the instructions:
>
>     "XXX The Sleepycat release we use will probably change before
>      XXX 2.3a1."
>
> and I didn't know whether that referred to a version upgrade or a
> complete replacement of the underlying DB engine with some other
> library, which would make any further work throwaway.

It refers to the current Sleepycat release; if you followed the build
instructions closely <wink>, they started by pointing you to a page of
obsolete releases.  For some reason Barry understands, we can't yet use the
current release (I think the bsddb C API changed, and our wrapper needs to
change accordingly).

Since I'm having a hard time (as above) imagining what good libdb40s.lib is
in its current state, I'm simply hoping that Sleepycat fixed the linking
glitches in their current release.

Thanks for the effort, Bob!  It helps.



From tim.one@comcast.net  Tue Dec  3 05:08:49 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 03 Dec 2002 00:08:49 -0500
Subject: [Python-Dev] RE: how to kill process on Windows started with
 os.spawn?
In-Reply-To: <15851.43071.25770.622477@montanaro.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEOHDAAB.tim.one@comcast.net>

[Tim]
> Why do you need to kill a process externally?  For example, why
> can't you make "please stop now" a part of the protocol,
> so that a process can terminate itself gracefully when told to?

[Skip Montanaro]
> I start the background ssh process from within pop3proxy, so it
> seems to me I ought to be the one to terminate it.

OK, that wasn't clear to me before.

> As for protocol, I'm not aware of another way to terminate a running
> ssh other than to call kill(2) with a specified signal.  That's
> more-or-less the Unix way.  I don't have the luxury of modifying
> ssh to understand some application-specific protocol.

No, I'm not asking you to be silly <wink>.  The Unix signal model isn't
going to fly for you on Windows regardless.



From martin@v.loewis.de  Tue Dec  3 07:41:23 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 08:41:23 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-4B275522065611D7ADDD003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-4B275522065611D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <m3el8zy2fw.fsf@mira.informatik.hu-berlin.de>

Just van Rossum <just@letterror.com> writes:

> - Allow the "Import from Zip archive" patch to be rewritten as an
> external module, avoiding the huge mess it is right now, as it
> touches more or less all of import.c and then some. Paul Moore is
> doing an admirable job to get it working and cleaned up, but I
> honestly don't see this getting into the core:

Actually, I do see it getting into the core, nearly as-is.

In what time frame would you expect to implement the zipfile import
patch using your framework?

Regards,
Martin



From martin@v.loewis.de  Tue Dec  3 08:17:52 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 09:17:52 +0100
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz>
References: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz>
Message-ID: <m365uby0r3.fsf@mira.informatik.hu-berlin.de>

Greg Ewing <greg@cosc.canterbury.ac.nz> writes:

> If this involves VPATH, I think I might know what's happening --
> something similar happened to me recently.
> 
> Have you by any chance previously done a non-VPATH build in the source
> tree?

I get the same problem when building in the source directory
(/usr/src/python). VPATH is not set in the Makefile, and the
compilation commands read like this

gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I. -I/usr/src/python/./Include -I/usr/local/include -I/usr/src/python/Include -I/usr/src/python -c /usr/src/python/Modules/regexmodule.c -o /usr/src/python/Modules/regexmodule.o

I believe this is triggered by the change

----------------------------
revision 1.45
date: 2001/08/05 22:31:19;  author: jackjansen;  state: Exp;  lines: +63 -6
Replace moddir and incdir by
moddirlist and incdirlist, lists of source and include
directories that are searched for modules.
This is needed because the Mac modules and include files
live in the Mac subtree.

In addition (and that's actually what the mod is all about) on OSX we build all the Mac extension modules.
----------------------------

Here, moddirlist is ['/usr/src/python/Modules'], so the name of the
source module will be /usr/src/python/Modules/regexmodule.c, instead of
Modules/regexmodule.c. In turn, build_ext.build_temp is ignored, and
the object files end up in the wrong place.

Jack, can you please correct this problem?

Regards,
Martin


From guido@python.org  Tue Dec  3 08:25:46 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 03:25:46 -0500
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: Your message of "Tue, 03 Dec 2002 17:48:55 +1300."
 <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz>
References: <200212030448.gB34mtx29036@kuku.cosc.canterbury.ac.nz>
Message-ID: <200212030825.gB38Pk101300@pcp02138704pcs.reston01.va.comcast.net>

> If this involves VPATH, I think I might know what's happening --
> something similar happened to me recently.
> 
> Have you by any chance previously done a non-VPATH build in the source
> tree?
> 
> It seems that, when using a VPATH, any pathname referencing something
> in the build directory for which a corresponding file exists in the
> VPATH directory gets changed to refer to the VPATH directory. So you
> need to make sure there are *only* source files in the source tree
> before doing a build using a VPATH.

This is indeed a common pitfall (== I've wasted an hour debugging this
once :-).  Can someone add a note to README?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Tue Dec  3 08:36:32 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 09:36:32 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <m3el8zy2fw.fsf@mira.informatik.hu-berlin.de>
Message-ID: <r01050400-1022-5721A128069A11D7ADDD003065D5E7E4@[10.0.0.23]>

Martin v. L=F6wis wrote:

> Actually, I do see it getting into the core, nearly as-is.

Ok, let's then say I'm opposed to the patch as it stand, as it messes up
import.c too much. The zip import stuff is _only_ in import.c because it'=
s
currently not possible to do it differently. This *must* be factored out,=
 and
now is the time. Don't you at least agree it's a hairy patch?

> In what time frame would you expect to implement the zipfile import
> patch using your framework?

Don't know, I can't do it by myself. Shouldn't be too hard, though.

Just


From just@letterror.com  Tue Dec  3 08:41:26 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 09:41:26 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <069901c299c2$d7988f80$0201010a@wiktor>
Message-ID: <r01050400-1022-0A53937B069B11D7ADDD003065D5E7E4@[10.0.0.23]>

Wiktor Sadowski wrote:

> Just van Rossum  wrote:
> 
> >>Could you please post a normal diff of your patch? It looks very
> >>interesting.
> Sorry, I haven't got time for cleaning up my modified Python C code
> and  building the diffs.
> 
> >.I'm not entirely sure yet how my idea and yours could be
> >integrated, but I think
> > it's work a look.
> 
> The concept is the same , there are differences in implementations.
> The implementation you have posted is much more pure Python oriented.
> I am using C structures and pointers to C functions (wherever
> possible) instead of Python objects (well PyObjects are still C
> structures) , so maybe it's a little faster (maybe not) 

I'm sure yours will be faster. However, the _same_ stuff is needed on the Python
end, and I think it can still be made fairly easy to do what you want with
sys.import_hooks. It can probably use a new C API to make things more
straightforward.

> Looks like I was wasting my time.;)

Not at all! If it wan't for your post I might not have implemented the
import_hooks stuff nearly as quickly, if at all ;-)

import.c is a mess, and I think my patch opens doors to actually clean it up
(perhaps for 2.4).

Just


From guido@python.org  Tue Dec  3 08:44:14 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 03:44:14 -0500
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: Your message of "Mon, 02 Dec 2002 12:43:01 CST."
 <15851.43445.443010.182362@montanaro.dyndns.org>
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net>
 <15851.43445.443010.182362@montanaro.dyndns.org>
Message-ID: <200212030844.gB38iEJ01481@pcp02138704pcs.reston01.va.comcast.net>

>     Guido> I'm still seeing errors in test_socket; today (on a different
>     Guido> network) I got this:
> 
>     Guido> test test_socket failed -- Traceback (most recent call last):
>     Guido>   File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes
>     Guido>     ip = socket.gethostbyname(hostname)
>     Guido> gaierror: (7, 'No address associated with nodename')
> 
> What does the hostname(1) command return?  By default I think it generates
> some oddball thing like "Montanaro.local.".  Apple calls it a Rendevous
> name.  (I've no idea what that is.)  The fix for me was recommended by
> another group: get a dyndns.org account and change the HOSTNAME line in
> /etc/hostconfig from
> 
>     HOSTNAME=-AUTOMATIC-
> 
> to
> 
>     HOSTNAME=montanaro.dyndns.org

Thanks, Skip, but I'm more interested in fixing test_socket so it
doesn 't rely on this.  Maybe testHostnameRes() should be disabled
when gethostbyaddr() gives an error?  It makes unwarranted assumptions
IMO about how DNS is set up.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Tue Dec  3 08:59:05 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 03:59:05 -0500
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: Your message of "Mon, 02 Dec 2002 17:33:25 EST."
 <ubs44vyoa.fsf@boost-consulting.com>
References: <003001c298d7$550e8240$0201010a@wiktor>
 <ubs44vyoa.fsf@boost-consulting.com>
Message-ID: <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net>

> My users want extension modules to be able to control how their
> attributes are imported when using "import *". In particular,
> Boost.Python includes a kind of multimethod facility, and some of my
> users would like it if
> 
>       from bpl_extension import *
> 
> caused multimethod definitions in bpl_extension to be /merged/ into
> the current module, rather than replacing any multimethod bindings
> with the same name in the current module.

Alas, the current import hooking mechanisms don't allow control over
this (the interpretation of * is hardcoded).  Feel free to suggest an
API and/or implementation of a hook for that, after reading how it's
done now.  The crucial code is import_all_from() in ceval.c.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From martin@v.loewis.de  Tue Dec  3 08:57:53 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 09:57:53 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-5721A128069A11D7ADDD003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-5721A128069A11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <m3of83wkby.fsf@mira.informatik.hu-berlin.de>

Just van Rossum <just@letterror.com> writes:

> Don't you at least agree it's a hairy patch?

It is very hard to read, as a patch, yes, and it does a lot of things
that aren't clearly needed.

However, in all cases I've investigated, the resulting import.c works
just as fine as the original one. Don't read the patch, read the
resulting code instead.

I don't like to reject the patch on grounds of "it is too
complicated". Perhaps the complexity is inherent, and it can't be done
trivially. In that case, it would be a pity since it means that we can
never get zipfile imports.

The feature that this patch provides is well-defined. If somebody
volunteers to cleanup the implementation later - this would be
definitely appreciated.

> > In what time frame would you expect to implement the zipfile import
> > patch using your framework?
> 
> Don't know, I can't do it by myself. Shouldn't be too hard, though.

That, of course, speaks against the framework, or atleast against
rejecting the zipfile patch.

It seems that the likeliness is high that we get a new framework, and
no application for it. If we require zipfile importing to use the new
framework, we will never get zipfile importing.

This doesn't sound very promising, so I'd mandate that any new
framework demonstrates its usefulness in a non-trivial case, which is
best done by implementing (or re-implementing) zipfile import.

Regards,
Martin


From just@letterror.com  Tue Dec  3 09:16:31 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 10:16:31 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <m3of83wkby.fsf@mira.informatik.hu-berlin.de>
Message-ID: <r01050400-1022-EB8BB7BC069F11D7ADDD003065D5E7E4@[10.0.0.23]>

Martin v. L=F6wis wrote:

> It is very hard to read, as a patch, yes, and it does a lot of things
> that aren't clearly needed.
>=20
> However, in all cases I've investigated, the resulting import.c works
> just as fine as the original one. Don't read the patch, read the
> resulting code instead.

And then it still does many things that "aren't clearly needed".

> I don't like to reject the patch on grounds of "it is too
> complicated". Perhaps the complexity is inherent, and it can't be done
> trivially.

The complexity is inherent *because* there's no import hook mechanism tha=
t's
even remotely usable from C. My patch changes that.

> In that case, it would be a pity since it means that we can
> never get zipfile imports.

No it doesn't. It just means that a little more work is needed.

> The feature that this patch provides is well-defined. If somebody
> volunteers to cleanup the implementation later - this would be
> definitely appreciated.

It doesn't *belong* in import.c. It's bloat.

> > > In what time frame would you expect to implement the zipfile import
> > > patch using your framework?
> >=20
> > Don't know, I can't do it by myself. Shouldn't be too hard, though.
>=20
> That, of course, speaks against the framework, or atleast against
> rejecting the zipfile patch.

I rather meant "I prefer not to do it by myself".

> It seems that the likeliness is high that we get a new framework, and
> no application for it. If we require zipfile importing to use the new
> framework, we will never get zipfile importing.

Bull.

> This doesn't sound very promising, so I'd mandate that any new
> framework demonstrates its usefulness in a non-trivial case, which is
> best done by implementing (or re-implementing) zipfile import.

...

Just


From martin@v.loewis.de  Tue Dec  3 09:31:53 2002
From: martin@v.loewis.de (Martin v. Löwis)
Date: Tue, 3 Dec 2002 10:31:53 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-EB8BB7BC069F11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <000d01c29aae$d1220590$6f21e8d9@mira>

> And then it still does many things that "aren't clearly needed".

Yes, but these are mostly reorganizations of stuff, eg. pieces of a
function moved into functions of their own. This is, in general, a good
thing, as it improves readability of the resulting code (not of the
patch, though).

> The complexity is inherent *because* there's no import hook mechanism
that's
> even remotely usable from C. My patch changes that.

I would expect that the complexity is inherent in "we implement our own
file system" (i.e. zipfiles). So I don't trust promises that a great new
framework will make things simpler, unless proven wrong, by seeing
actual things that become actually simpler.

> No it doesn't. It just means that a little more work is needed.

The current patch was sitting on SF for almost a year, and nobody even
looked at it. How many more years do you want to wait for somebody
finding time for "a little more work"?

> It doesn't *belong* in import.c. It's bloat.

I don't understand. import.c does the implementation of the Python
import statement, and the patch extends the semantics of the import
statement, by allowing .zip files to be on sys.path. You could argue
that the feature itself is bloat, but if the feature is desirable, I
can't see how the implementation is bloat.

> I rather meant "I prefer not to do it by myself".

I can understand that. Unfortunately, past experience tells me that
nobody will do it, then.

If more people speak in favour of rejecting the zipfile import patch, we
should probably tell Paul Moore, so that won't invest more time into
what is a doomed project.

Regards,
Martin



From guido@python.org  Tue Dec  3 09:52:35 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 04:52:35 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Tue, 03 Dec 2002 10:31:53 +0100."
 <000d01c29aae$d1220590$6f21e8d9@mira>
References: <r01050400-1022-EB8BB7BC069F11D7ADDD003065D5E7E4@[10.0.0.23]>
 <000d01c29aae$d1220590$6f21e8d9@mira>
Message-ID: <200212030952.gB39qZA02315@pcp02138704pcs.reston01.va.comcast.net>

> If more people speak in favour of rejecting the zipfile import patch, we
> should probably tell Paul Moore, so that won't invest more time into
> what is a doomed project.

I agree with both sides here. :-( / :-)

The zipfile feature is important, and if this patch is the only way to
get the feature in, so be it.

But import.c really can use a cleanup.

I wouldn't be against incorporating some version of the patch now, and
cleaning it up later, possibly adding a new import hooking framework
and moving some of the zip specific code out of import.c again.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mwh@python.net  Tue Dec  3 10:08:52 2002
From: mwh@python.net (Michael Hudson)
Date: 03 Dec 2002 10:08:52 +0000
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: Skip Montanaro's message of "Mon, 2 Dec 2002 22:29:09 -0600"
References: <15852.13077.577684.9372@montanaro.dyndns.org>
Message-ID: <2misybjtxn.fsf@starship.python.net>

Skip Montanaro <skip@pobox.com> writes:

> I don't have time to look into this right now.  I only just noticed it while
> I was trying to do parallel builds in build.dbg and build.opt directories.
> The debug .o files got written first, so when the optimized make came along,
> there were already .o files, which had been compiled with --with-pydebug,
> got linked improperly.

Ah, this would be why I couldn't build a debug python on my iBook, then.

If this is Jack's fault, as Martin suggests, we should bully him until
he fixes it!

Cheers,
M.

-- 
  > so python will fork if activestate starts polluting it?
  I find it more relevant to speculate on whether Python would fork
  if the merpeople start invading our cities riding on the backs of 
  giant king crabs.                 -- Brian Quinlan, comp.lang.python


From mal@lemburg.com  Tue Dec  3 10:28:13 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 03 Dec 2002 11:28:13 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-EB8BB7BC069F11D7ADDD003065D5E7E4@[10.0.0.23]> <000d01c29aae$d1220590$6f21e8d9@mira>
Message-ID: <3DEC873D.7020406@lemburg.com>

Martin v. L=F6wis wrote:
> If more people speak in favour of rejecting the zipfile import patch, w=
e
> should probably tell Paul Moore, so that won't invest more time into
> what is a doomed project.

I have been waiting for such a patch to get incorporated for a long
time now. ZIP file imports would be great for packaging software
in a way which lets the packager have more control of what's lying
around on the user machine, e.g. it allows removing cruft very
easily and makes user space tampering harder.

Whatever you come up with, please make sure that ZIP file based
imports become possible.

Thanks,
--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From just@letterror.com  Tue Dec  3 10:37:51 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 11:37:51 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <000d01c29aae$d1220590$6f21e8d9@mira>
Message-ID: <r01050400-1022-4A867C2A06AB11D7ADDD003065D5E7E4@[10.0.0.23]>

Martin v. L=F6wis wrote:

> > And then it still does many things that "aren't clearly needed".
>=20
> Yes, but these are mostly reorganizations of stuff, eg. pieces of a
> function moved into functions of their own. This is, in general, a good
> thing, as it improves readability of the resulting code (not of the
> patch, though).

Besides bloat, my main objection to the patch is that it refactors import=
.c in a
way that's not general enough and actually make import.c more complex tha=
n
needed. This is IMO a bad thing.

> > The complexity is inherent *because* there's no import hook mechanism
> > that's even remotely usable from C. My patch changes that.
>=20
> I would expect that the complexity is inherent in "we implement our own
> file system" (i.e. zipfiles). So I don't trust promises that a great ne=
w
> framework will make things simpler, unless proven wrong, by seeing
> actual things that become actually simpler.

True. See below.

> > No it doesn't. It just means that a little more work is needed.
>=20
> The current patch was sitting on SF for almost a year, and nobody even
> looked at it. How many more years do you want to wait for somebody
> finding time for "a little more work"?

It's been sitting there for so long *because* it's such a hairy patch. I'=
ve just
started writing a new madule called "zipimport" which will show what I me=
an. The
rest of today will be very busy here, so you _might_ have to wait until t=
omorrow
for working code.

Just


From fredrik@pythonware.com  Tue Dec  3 11:34:21 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Tue, 3 Dec 2002 12:34:21 +0100
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>              <077101c29a13$a9b78cf0$0900a8c0@spiff>  <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <0ad001c29abf$ed602f50$0900a8c0@spiff>

guido wrote:


> >     try:
> >         import threading
> >     except ImportError:
> >         threading =3D None # use non-threaded behaviour
>=20
> But then you'd have to sprinkle the rest of the code with
> "if threading is not None" tests.  Brett's dummy_thread is an attempt
> to reduce this clutter.

the problem (if there is one) is that the proposed integration may
break code that uses the above pattern.

deadlocks doesn't always happen when you want them to, so it
may not be obvious to developers and testers that their program
isn't as reliable as it was before the upgrade.

</F>



From Paul.Moore@atosorigin.com  Tue Dec  3 11:48:29 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Tue, 3 Dec 2002 11:48:29 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619969@UKDCX001.uk.int.atosorigin.com>

(I'm reading python-dev via the web, so this is a combined reply
to various posts)

First of all, let me say that I'm interested in imports from
zipfiles, as a feature. I don't have any intrinsic reason to
prefer a specific implementation over any other. I picked up Jim
Ahlstrom's patch and had a look at it simply because it seemed
the best way to help.

My view is that import.c is, indeed, very complex and difficult
code to follow. I haven't made a significant attempt to do so,
taking the view that as Jim's patch basically worked for him, I
will just look at updating things to cope with the changes in
CVS since his patch was current. (The comments in SF indicate
that the patch was nearly accepted as it stood, and just needed
updating to match current CVS).

I agree with Just's comments that a cleanup in import.c would be
useful. But I'm not in a position to do that, much as I would
like it. (One example - "universal newline" support could do with
being better localised - I just implemented the 4th copy of a
tiny chunk of code to juggle file open modes to support it).

I think that the zipfile patch is close to being acceptable. I
don't have a problem if it gets rejected, but can I make a plea
- if it does, can it be "Rejected because patch XXX does the job
better" rather than "Rejected because this can be implemented
better using feature YYY". In other words, as Martin suggests,
can any competing proposal actually *implement* zipfile imports,
so that we don't hit a new delay.

By the way, the zipfile import also implements a cacheing feature
for directory lookups. PEP 273 gives more details, including
some benchmarks, but the indications are that the cache speeds
up normal imports as well as being crucial for zipfile imports
(where directory lookups are *slow*). I can't see how the design
of Just's patch would help us implement this cache - maybe he can
comment.

Finally, while we're looking at imports and hooks, has anyone
considered Gordon McMillan's imputil replacement, as described at
http://www.mcmillan-inc.com/iu.html? I've used it to implement a
proof-of-concept zipfile import (in Python) before now, and it
was pretty simple to use. Maybe this would be a good candidate
for the standard library?

Thanks to everyone for the interest and support.
Paul.


From guido@python.org  Tue Dec  3 12:07:53 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 07:07:53 -0500
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: Your message of "Tue, 03 Dec 2002 12:34:21 +0100."
 <0ad001c29abf$ed602f50$0900a8c0@spiff>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU> <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>
 <0ad001c29abf$ed602f50$0900a8c0@spiff>
Message-ID: <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>

> > >     try:
> > >         import threading
> > >     except ImportError:
> > >         threading = None # use non-threaded behaviour
> > 
> > But then you'd have to sprinkle the rest of the code with
> > "if threading is not None" tests.  Brett's dummy_thread is an attempt
> > to reduce this clutter.
> 
> the problem (if there is one) is that the proposed integration may
> break code that uses the above pattern.

I don't propose to automate this.  I propose this:

  try:
    import threading
  except ImportError:
    import dummy_threading as threading

> deadlocks doesn't always happen when you want them to, so it
> may not be obvious to developers and testers that their program
> isn't as reliable as it was before the upgrade.

So require testing on a system that has threads.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@python.org  Tue Dec  3 13:20:07 2002
From: barry@python.org (Barry A. Warsaw)
Date: Tue, 3 Dec 2002 08:20:07 -0500
Subject: [Python-Dev] Mac OSX issues
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net>
 <15850.58353.356475.182137@gargle.gargle.HOWL>
 <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net>
 <15851.43445.443010.182362@montanaro.dyndns.org>
 <200212030844.gB38iEJ01481@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15852.44935.756518.645276@gargle.gargle.HOWL>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> Thanks, Skip, but I'm more interested in fixing test_socket
    GvR> so it doesn 't rely on this.  Maybe testHostnameRes() should
    GvR> be disabled when gethostbyaddr() gives an error?  It makes
    GvR> unwarranted assumptions IMO about how DNS is set up.

BTW, test_socket.py passes just fine for me.  hostname(1) returns
"freewill.local." (yes, trailing dot).

The difference may be that I used Netinfo Manager to add hostname
entries for all my local machines, including the MacOSX machine I ran
the tests on.  My Netinfo-fu is pretty meager so I don't know if I did
it completely right, but it seems to work for me.

-Barry


From sholden@holdenweb.com  Tue Dec  3 13:28:46 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Tue, 3 Dec 2002 08:28:46 -0500
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-EB8BB7BC069F11D7ADDD003065D5E7E4@[10.0.0.23]>              <000d01c29aae$d1220590$6f21e8d9@mira>  <200212030952.gB39qZA02315@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <009d01c29acf$eab0d380$6300000a@holdenweb.com>

[MvL]
> > If more people speak in favour of rejecting the zipfile import patch, we
> > should probably tell Paul Moore, so that won't invest more time into
> > what is a doomed project.
[Guido]
>
> I agree with both sides here. :-( / :-)
>
> The zipfile feature is important, and if this patch is the only way to
> get the feature in, so be it.
>
> But import.c really can use a cleanup.
>
> I wouldn't be against incorporating some version of the patch now, and
> cleaning it up later, possibly adding a new import hooking framework
> and moving some of the zip specific code out of import.c again.
>

Agreed. Like MAL I feel this is an important patch to get in to the
distribution, even if it leaves a refactoring job behind it. Safe for me to
say when I no longer do C, I suppose :-) but I'm speaking from a user
perspective here.

regards
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Previous .sig file retired to                    www.homeforoldsigs.com
-----------------------------------------------------------------------





From jepler@unpythonic.net  Tue Dec  3 14:24:20 2002
From: jepler@unpythonic.net (Jeff Epler)
Date: Tue, 3 Dec 2002 08:24:20 -0600
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net>
References: <003001c298d7$550e8240$0201010a@wiktor> <ubs44vyoa.fsf@boost-consulting.com> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20021203142419.GC2507@unpythonic.net>

On Tue, Dec 03, 2002 at 03:59:05AM -0500, Guido van Rossum wrote:
> Alas, the current import hooking mechanisms don't allow control over
> this (the interpretation of * is hardcoded).  Feel free to suggest an
> API and/or implementation of a hook for that, after reading how it's
> done now.  The crucial code is import_all_from() in ceval.c.

Currently __all__ is a list of all the attributes to be exported by the
module.  What if we let __all__ be a list (with its current
interpretation) or a callable.

If __all__ is callable, then it is called with two parameters: the
importing module, and the imported module.  It can perform the desired
operation.  Any returned value is ignored, any raised exception is
propagated.

In David Abraham's case, the hook might look something like this:

    def __all__(exporter, importer):
        for name, attr in vars(exporter).items():
            if name.startswith('_'): continue
            if isinstance(attr, Multimethod):
                merge_one_multimethod(importer, name, attr)
            else:
                setattr(importer, name, method)

(David, apologies if I've simplified this too much.  All I know about
your desired functionality is that you want to "merge multimethods", and
I have some idea what a multimethod is..)

Jeff


From neal@metaslash.com  Tue Dec  3 14:28:05 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 3 Dec 2002 09:28:05 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619969@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619969@UKDCX001.uk.int.atosorigin.com>
Message-ID: <20021203142805.GC16807@epoch.metaslash.com>

On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote:
> 
> My view is that import.c is, indeed, very complex and difficult
> code to follow. 
> I agree with Just's comments that a cleanup in import.c would be
> useful. 

I agree with both of these statements.  I have reviewed the patch.
While it's quite involved, I don't think it's worth rejecting.
I would really like to see a cleanup, though.

Rejecting the patch would be worthwhile if the interface would need to
change after incorporating this patch.  I don't believe this is the
case.  Whether this patch is accepted or not wouldn't change future
patches that Just wants to incorporate.

> I think that the zipfile patch is close to being acceptable. I
> don't have a problem if it gets rejected, but can I make a plea
> - if it does, can it be "Rejected because patch XXX does the job
> better" rather than "Rejected because this can be implemented
> better using feature YYY".

Agreed.

I don't recall if there is a lot of code duplication for file handling
in import.c, there's certainly some.  But in reviewing bz2, there was
a lot of duplication.  Gustavo and I discussed reusing the file code.
Perhaps, PyFileObject could be refactored to make reuse easier in bz2
and import.

If Just incorporates his import patch, perhaps the zipfile code could
be refactored to use the more generic framework.  I don't view these
two patches as mutually exclusive.  If all of these issues are
addressed, the resulting code could be clearer, even if intermediate
versions are not.

Neal



From mwh@python.net  Tue Dec  3 14:31:15 2002
From: mwh@python.net (Michael Hudson)
Date: 03 Dec 2002 14:31:15 +0000
Subject: [Python-Dev] New universal import mechanism ( Re:  [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: David Abrahams's message of "Mon, 02 Dec 2002 17:33:25 -0500"
References: <003001c298d7$550e8240$0201010a@wiktor> <ubs44vyoa.fsf@boost-consulting.com>
Message-ID: <2mfztfjhsc.fsf@starship.python.net>

David Abrahams <dave@boost-consulting.com> writes:

> My users want extension modules to be able to control how their
> attributes are imported when using "import *". In particular,
> Boost.Python includes a kind of multimethod facility, and some of my
> users would like it if
> 
>       from bpl_extension import *
> 
> caused multimethod definitions in bpl_extension to be /merged/ into
> the current module, rather than replacing any multimethod bindings
> with the same name in the current module.

Eww.  I can see where the idea is coming from, but allowing arbitrary
magic somewhere only name binding is allowed today sounds horrible to
me.

Cheers,
M.

-- 
  please realize that the Common  Lisp community is more than 40
  years old.  collectively, the community has already been where
  every clueless newbie  will be going for the next three years.
  so relax, please.                     -- Erik Naggum, comp.lang.lisp


From pedronis@bluewin.ch  Tue Dec  3 14:24:59 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 15:24:59 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
References: <003001c298d7$550e8240$0201010a@wiktor> <ubs44vyoa.fsf@boost-consulting.com> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net>
Message-ID: <00a701c29ad7$c33b8860$6d94fea9@newmexico>

----- Original Message -----
From: "Jeff Epler" <jepler@unpythonic.net>
To: <python-dev@python.org>
Sent: Tuesday, December 03, 2002 3:24 PM
Subject: Re: [Python-Dev] New universal import mechanism ( Re:
[Python-checkins] python/dist/src/Python import.c,2.210,2.211)


> On Tue, Dec 03, 2002 at 03:59:05AM -0500, Guido van Rossum wrote:
> > Alas, the current import hooking mechanisms don't allow control over
> > this (the interpretation of * is hardcoded).  Feel free to suggest an
> > API and/or implementation of a hook for that, after reading how it's
> > done now.  The crucial code is import_all_from() in ceval.c.
>
> Currently __all__ is a list of all the attributes to be exported by the
> module.  What if we let __all__ be a list (with its current
> interpretation) or a callable.
>
> If __all__ is callable, then it is called with two parameters: the
> importing module, and the imported module.  It can perform the desired
> operation.  Any returned value is ignored, any raised exception is
> propagated.
>
> In David Abraham's case, the hook might look something like this:
>
>     def __all__(exporter, importer):
>         for name, attr in vars(exporter).items():
>             if name.startswith('_'): continue
>             if isinstance(attr, Multimethod):
>                 merge_one_multimethod(importer, name, attr)
>             else:
>                 setattr(importer, name, method)
>
> (David, apologies if I've simplified this too much.  All I know about
> your desired functionality is that you want to "merge multimethods", and
> I have some idea what a multimethod is..)
>

I find this idea of a semantics change rather unpythonic;
( 2nd in general should  not  import * be avoided ?)

Why then only import * should have a user-defined behavior?

What about

from module import x

if the importing module has already a defined x then maybe that should be also
under user control?

I'm surely missing something.

regards.



From dave@boost-consulting.com  Tue Dec  3 14:40:43 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Tue, 03 Dec 2002 09:40:43 -0500
Subject: [Python-Dev] New universal import mechanism ( Re:
 [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <20021203142419.GC2507@unpythonic.net> (Jeff Epler's message of
 "Tue, 3 Dec 2002 08:24:20 -0600")
References: <003001c298d7$550e8240$0201010a@wiktor>
 <ubs44vyoa.fsf@boost-consulting.com>
 <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net>
 <20021203142419.GC2507@unpythonic.net>
Message-ID: <uvg2bi2s4.fsf@boost-consulting.com>

Jeff Epler <jepler@unpythonic.net> writes:

> On Tue, Dec 03, 2002 at 03:59:05AM -0500, Guido van Rossum wrote:
>> Alas, the current import hooking mechanisms don't allow control over
>> this (the interpretation of * is hardcoded).  Feel free to suggest an
>> API and/or implementation of a hook for that, after reading how it's
>> done now.  The crucial code is import_all_from() in ceval.c.
>
> Currently __all__ is a list of all the attributes to be exported by the
> module.  What if we let __all__ be a list (with its current
> interpretation) or a callable.
>
> If __all__ is callable, then it is called with two parameters: the
> importing module, and the imported module.  It can perform the desired
> operation.  Any returned value is ignored, any raised exception is
> propagated.
>
> In David Abraham's case, the hook might look something like this:
>
>     def __all__(exporter, importer):
>         for name, attr in vars(exporter).items():
>             if name.startswith('_'): continue
>             if isinstance(attr, Multimethod):
>                 merge_one_multimethod(importer, name, attr)
>             else:
>                 setattr(importer, name, method)
>
> (David, apologies if I've simplified this too much.  All I know about
> your desired functionality is that you want to "merge multimethods", and
> I have some idea what a multimethod is..)

Nice; I think this pretty much captures what my users are after.

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From just@letterror.com  Tue Dec  3 14:52:18 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 15:52:18 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <20021203142805.GC16807@epoch.metaslash.com>
Message-ID: <r01050400-1022-DFEE7F9606CE11D7ADDD003065D5E7E4@[10.0.0.23]>

Neal Norwitz wrote:

> On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote:
> > 
> > My view is that import.c is, indeed, very complex and difficult
> > code to follow. I agree with Just's comments that a cleanup in
> > import.c would be useful. 
> 
> I agree with both of these statements.  I have reviewed the patch.
> While it's quite involved, I don't think it's worth rejecting.
> I would really like to see a cleanup, though.

I'm working hard on a zipimport module that uses sys.import_hooks. It's going to
be SO much cleaner than the existing zip import patch.

> Rejecting the patch would be worthwhile if the interface would need to
> change after incorporating this patch.  I don't believe this is the
> case.  Whether this patch is accepted or not wouldn't change future
> patches that Just wants to incorporate.

I personally reject the patch because it modifies import.c beyond repair. Once
this stuff is in, there's NO way we can go back without a huge effort, meaning
it won't happen. It adds a whole directory listing caching scheme that is a)
totally unneeded and b) has (IMO) _nothing_ to do with zip importing per se.
That ExternalNames/External names stuff? I don't understand why it is needed
just to enable zip imports. It's a lot of stuff!

> > I think that the zipfile patch is close to being acceptable. I
> > don't have a problem if it gets rejected, but can I make a plea
> > - if it does, can it be "Rejected because patch XXX does the job
> > better" rather than "Rejected because this can be implemented
> > better using feature YYY".
> 
> Agreed.

And that's ecxactly what I intend to say in a couple of days if not sooner.

[ ... ]
> If Just incorporates his import patch, perhaps the zipfile code could
> be refactored to use the more generic framework.  I don't view these
> two patches as mutually exclusive. 

I *totally* do.

> If all of these issues are
> addressed, the resulting code could be clearer, even if intermediate
> versions are not.

A rewrite now is MUCH easier than cleaning it up later. It's going to be much
less code. Wait and see.

Just


From dave@boost-consulting.com  Tue Dec  3 14:45:24 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Tue, 03 Dec 2002 09:45:24 -0500
Subject: [Python-Dev] New universal import mechanism ( Re:
 [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <00a701c29ad7$c33b8860$6d94fea9@newmexico> ("Samuele Pedroni"'s
 message of "Tue, 3 Dec 2002 15:24:59 +0100")
References: <003001c298d7$550e8240$0201010a@wiktor>
 <ubs44vyoa.fsf@boost-consulting.com>
 <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net>
 <20021203142419.GC2507@unpythonic.net>
 <00a701c29ad7$c33b8860$6d94fea9@newmexico>
Message-ID: <ur8czi2kb.fsf@boost-consulting.com>

"Samuele Pedroni" <pedronis@bluewin.ch> writes:

> From: "Jeff Epler" <jepler@unpythonic.net>
>
>> Currently __all__ is a list of all the attributes to be exported by the
>> module.  What if we let __all__ be a list (with its current
>> interpretation) or a callable.

<snip>

> I find this idea of a semantics change rather unpythonic;

Of course you do; look at Jeff's email address! Wish I'd thought of
establishing that one myself <wink>. Calling something "un-pythonic"
is a very convenient form of derision.

But seriously: it's not a semantics change, but an extension... unless
you think people are making __all__ callable today, and counting on an
exception from the import mechanism.

> ( 2nd in general should  not  import * be avoided ?)
>
> Why then only import * should have a user-defined behavior?
>
> What about
>
> from module import x

I had the same thoughts. I am not entirely convinced that what my user
has asked for is the best approach.

> if the importing module has already a defined x then maybe that
> should be also under user control?

I don't understand that suggestion. Care to elaborate?

> I'm surely missing something.

it ain't neccessarily so...

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From pedronis@bluewin.ch  Tue Dec  3 14:41:14 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 15:41:14 +0100
Subject: [Python-Dev] New universal import mechanism ( Re:  [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
References: <003001c298d7$550e8240$0201010a@wiktor> <ubs44vyoa.fsf@boost-consulting.com> <2mfztfjhsc.fsf@starship.python.net>
Message-ID: <00ef01c29ada$083da180$6d94fea9@newmexico>

From: "Michael Hudson" <mwh@python.net>
> 
> Eww.  I can see where the idea is coming from, but allowing arbitrary
> magic somewhere only name binding is allowed today sounds horrible to
> me.
> 

Indeed, that's exactly my concern too, and that then it should consider all
from import forms.


From pedronis@bluewin.ch  Tue Dec  3 14:48:12 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 15:48:12 +0100
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
References: <003001c298d7$550e8240$0201010a@wiktor><ubs44vyoa.fsf@boost-consulting.com><200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net><20021203142419.GC2507@unpythonic.net><00a701c29ad7$c33b8860$6d94fea9@newmexico> <ur8czi2kb.fsf@boost-consulting.com>
Message-ID: <011f01c29adb$01033d20$6d94fea9@newmexico>

From: "David Abrahams" <dave@boost-consulting.com>
> "Samuele Pedroni" <pedronis@bluewin.ch> writes:
...
> > What about
> >
> > from module import x
>
> I had the same thoughts. I am not entirely convinced that what my user
> has asked for is the best approach.
>
> > if the importing module has already a defined x then maybe that
> > should be also under user control?
>
> I don't understand that suggestion. Care to elaborate?
>

my point is that if you offer a way to hook into the behavior of from import *
then you need for completeness and to establish a coherent whole to offer a
hook also in

from import foo,bar

regards.



From Paul.Moore@atosorigin.com  Tue Dec  3 15:47:42 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Tue, 3 Dec 2002 15:47:42 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com>

From: Just van Rossum [mailto:just@letterror.com]

> Neal Norwitz wrote:
>
> > On Tue, Dec 03, 2002 at 11:48:29AM -0000, Moore, Paul wrote:
> >
> > > My view is that import.c is, indeed, very complex and
> > > difficult code to follow. I agree with Just's comments that
> > > a cleanup in import.c would be useful.
> >
> > I agree with both of these statements. I have reviewed the
> > patch. While it's quite involved, I don't think it's worth
> > rejecting I would really like to see a cleanup, though
>
> I'm working hard on a zipimport module that uses
> sys.import_hooks. It's going to be SO much cleaner than the
> existing zip import patch.

That's great news, as I've said before. Can I ask that you
consider a few issues, though. I don't have particular use cases
in mind, but I believe that PEP 273 had a number of implicit
assumptions, which your proposal does not incorporate. I'd like
to make the differences explicit, just so that people get a
chance to comment on whether they care.

1. Your import hooks seem to act at a level "above" sys.path,
   in the sense that the normal sys.path search becomes just
   another hook. With that in mind, I don't see a way for a user
   to add a zipfile in the *middle* of sys.path.
2. The syntax needed to add a zipfile search is different from
   the familiar one for sys.path additions. This isn't an issue
   as such, but may make the feature a little less accessible.
3. By using a non-sys.path approach, you make it impossible for
   users to set PYTHONPATH to include zipfiles.

Can I again ask you to review Gordon McMillan's import manager.
I really feel that you're reinventing the wheel here, and Gordon's
code is well tested, as it's part of his Installer package.

Maybe Gordon's code (which is pure Python) isn't appropriate to
use, but I'm sure the design is worth looking at.

> > Rejecting the patch would be worthwhile if the interface
> > would need to change after incorporating this patch. I don't
> > believe this is the case. Whether this patch is accepted
> > or not wouldn't change future patches that Just wants to
> > incorporate.
>
> I personally reject the patch because it modifies import.c
> beyond repair. Once this stuff is in, there's NO way we can go
> back without a huge effort, meaning it won't happen.

I think you're exaggerating. It makes things worse, certainly,
but I'd strongly dispute your claim of "beyond repair". I simply
don't think the change is that major. Either it's already beyond
repair, or the patched version is still fixable.

And nobody is proposing "going back". A refactoring is "going
forward". Whether it's a refactoring of code that *already*
supports zipfiles or a refactoring which then has to be changed
to add zipfile support shouldn't matter.

> It adds
> a whole directory listing caching scheme that is a) totally
> unneeded and b) has (IMO) _nothing_ to do with zip importing
> per se. That ExternalNames/External names stuff? I don't
> understand why it is needed just to enable zip imports. It's a
> lot of stuff!

I've explained this a couple of times now. Please read the PEP
for details, but Jim claims that zipfile imports need this if
they are not to be too slow. A directory lookup in a zipfile is
basically a linear search through the list of all files in the
zipfile. For a large zipfile (such as the Python library) this
is going to be *slow*.

Please benchmark your implementation on a zipped version of the
full Python library, replacing the directory version, and compare
with Jim's benchmarks. While speed isn't the be-all and end-all
of this patch, I'd hate to lose a significant speed gain...

And the cache has a nice side-effect in that it speeds up general
imports, particularly on networked drives.

> > > I think that the zipfile patch is close to being
> > > acceptable. I don't have a problem if it gets rejected, but
> > > can I make a plea - if it does, can it be "Rejected because
> > > patch XXX does the job better" rather than "Rejected
> > > because this can be implemented better using feature YYY".
> >
> > Agreed.
>
> And that's ecxactly what I intend to say in a couple of days if
> not sooner.

Great! Will your code address all of the above points, and the
issues raised on python-dev in Oct/Nov 2001 (see below for
references)?

> [ ... ]
> > If Just incorporates his import patch, perhaps the zipfile
> > code could be refactored to use the more generic framework. I
> > don't view these two patches as mutually exclusive.
>
> I *totally* do.
>
> > If all of these issues are addressed, the resulting code
> > could be clearer, even if intermediate versions are not.
>
> A rewrite now is MUCH easier than cleaning it up later. It's
> going to be much less code. Wait and see.

Here are some references that anyone looking at this stuff should
read. There was a *lot* of discussion on this patch in Oct/Nov 2001.

Start of the PEP 273 thread:
http://mail.python.org/pipermail/python-dev/2001-October/018158.html

Thread on caching of lookups
http://mail.python.org/pipermail/python-dev/2001-November/018346.html

Thomas Heller proposing Gordon McMillan's code for the standard
library:
http://mail.python.org/pipermail/python-dev/2001-October/017750.html

As I've said, I'm not against rejecting the current patch - but I'd
be concerned if all of the work that went into it (ie, Jim's design
and discussions on python-dev, *not* the specific code implementing
it) was just lost.

Paul.


From jepler@unpythonic.net  Tue Dec  3 16:03:09 2002
From: jepler@unpythonic.net (Jeff Epler)
Date: Tue, 3 Dec 2002 10:03:09 -0600
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <00a701c29ad7$c33b8860$6d94fea9@newmexico>
References: <003001c298d7$550e8240$0201010a@wiktor> <ubs44vyoa.fsf@boost-consulting.com> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net> <00a701c29ad7$c33b8860$6d94fea9@newmexico>
Message-ID: <20021203160258.GD2507@unpythonic.net>

On Tue, Dec 03, 2002 at 03:24:59PM +0100, Samuele Pedroni wrote:
> Why then only import * should have a user-defined behavior?
> 
> What about
> 
> from module import x
> 
> if the importing module has already a defined x then maybe that should be also
> under user control?
> 
> I'm surely missing something.

I don't have a good sense of why this feature is desirable, I was just
suggesting how David's request might be implemented without introducing
new __special__ names.

If it was desired that 'from module import x' be able to use the special
behavior, then I'd suggest leaving __all__ as it is now, and having
__export__, with the default behaving about like this:

    def __export__(exporter, importer, names):
        for name in names:
            setattr(importer, name, getattr(exporter, name))

and David's __export__ being about like this:

    def __export__(exporter, importer, names):
        for name in names:
            attr = getattr(exporter, name)
            if isinstance(attr, Multimethod):
                merge_one_multimethod(exporter, name, attr)
            else:
                setattr(exporter, name, attr)

maybe __export__ should be called once per name, or maybe a special
value should be passed for * (instead of __all__)

Of course, 'from m import *' has that "do with me what you will" feeling
to it, while 'from m import x' doesn't.

Jeff


From jepler@unpythonic.net  Tue Dec  3 16:08:28 2002
From: jepler@unpythonic.net (Jeff Epler)
Date: Tue, 3 Dec 2002 10:08:28 -0600
Subject: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
In-Reply-To: <20021203160258.GD2507@unpythonic.net>
References: <003001c298d7$550e8240$0201010a@wiktor> <ubs44vyoa.fsf@boost-consulting.com> <200212030859.gB38x5U01600@pcp02138704pcs.reston01.va.comcast.net> <20021203142419.GC2507@unpythonic.net> <00a701c29ad7$c33b8860$6d94fea9@newmexico> <20021203160258.GD2507@unpythonic.net>
Message-ID: <20021203160827.GE2507@unpythonic.net>

small change to example code

On Tue, Dec 03, 2002 at 10:03:09AM -0600, Jeff Epler wrote:
> and David's __export__ being about like this:
> 
>     def __export__(exporter, importer, names):
>         for name in names:
>             attr = getattr(exporter, name)
>             if isinstance(attr, Multimethod):
-                 merge_one_multimethod(exporter, name, attr)
+                 merge_one_multimethod(importer, name, attr)
>             else:
>                 setattr(exporter, name, attr)


From pedronis@bluewin.ch  Tue Dec  3 16:05:24 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 17:05:24 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com>
Message-ID: <016e01c29ae5$c9ec73a0$6d94fea9@newmexico>

From: "Moore, Paul" <Paul.Moore@atosorigin.com>

>That's great news, as I've said before. Can I ask that you
>consider a few issues, though. I don't have particular use cases
>in mind, but I believe that PEP 273 had a number of implicit
>assumptions, which your proposal does not incorporate. I'd like
>to make the differences explicit, just so that people get a
>chance to comment on whether they care.
>
>1. Your import hooks seem to act at a level "above" sys.path,
>   in the sense that the normal sys.path search becomes just
>   another hook. With that in mind, I don't see a way for a user
>   to add a zipfile in the *middle* of sys.path.

It is worth to notice that Jython 2.1 already implement more or less PEP 273
approach, allowing to add zipfiles to sys.path.

We can live with a different/new approach and then plan a migration path with
warning etc, but if that could be avoided it would be nice.

regards.



From just@letterror.com  Tue Dec  3 16:39:36 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 17:39:36 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-D28FAF6406DD11D7ADDD003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> That's great news, as I've said before. Can I ask that you
> consider a few issues, though. I don't have particular use cases
> in mind, but I believe that PEP 273 had a number of implicit
> assumptions, which your proposal does not incorporate. I'd like
> to make the differences explicit, just so that people get a
> chance to comment on whether they care.
> 
> 1. Your import hooks seem to act at a level "above" sys.path,
>    in the sense that the normal sys.path search becomes just
>    another hook. With that in mind, I don't see a way for a user
>    to add a zipfile in the *middle* of sys.path.

True. There would be ways around that, but it's currently not easy. I don't feel
this is a particularly important feature, though.

> 2. The syntax needed to add a zipfile search is different from
>    the familiar one for sys.path additions. This isn't an issue
>    as such, but may make the feature a little less accessible.

The way I see it: sys.path is for file system imports. It's a list of
directories, nothing else.

> 3. By using a non-sys.path approach, you make it impossible for
>    users to set PYTHONPATH to include zipfiles.

This could be fixed by site.py: it could remove any .zip files from sys.path and
add hooks for them in sys.import_hooks. Or it could be fixed in the C code that
adds the script's directory to sys.path. Or we could add a tiny change to
import.c that invokes the zipimport module if a zip file is encountered.

[ snip ]
> > It adds
> > a whole directory listing caching scheme that is a) totally
> > unneeded and b) has (IMO) _nothing_ to do with zip importing
> > per se. That ExternalNames/External names stuff? I don't
> > understand why it is needed just to enable zip imports. It's a
> > lot of stuff!
> 
> I've explained this a couple of times now. Please read the PEP
> for details, but Jim claims that zipfile imports need this if
> they are not to be too slow.

Well, I am not going to need it: upon reading the zip contents table I build a
dict, mapping (full) module names to toc info. One dict lookup will be enough to
get at the right data.

Let's discuss the rest when my implementation actually works.

Just


From Paul.Moore@atosorigin.com  Tue Dec  3 16:48:14 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Tue, 3 Dec 2002 16:48:14 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E4A@UKDCX001.uk.int.atosorigin.com>

From: Just van Rossum [mailto:just@letterror.com]
> > I've explained this a couple of times now. Please read
> > the PEP for details, but Jim claims that zipfile imports
> > need this if they are not to be too slow.

> Well, I am not going to need it: upon reading the zip
> contents table I build a dict, mapping (full) module
> names to toc info. One dict lookup will be enough to
> get at the right data.

But that's exactly what Jim's cache is doing. His code is
general, and offers the service to *all* import methods,
rather than restricting it to only zipfiles. His benchmarks
showed that this was a (small) win.

Perhaps you could refactor your implementation so that the
cache isn't just for zipfiles? (I can easily imagine other
extensions that would benefit - tarfiles, metakit databases,
imports over the web, ...)

Paul.


From skip@pobox.com  Tue Dec  3 16:52:26 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 3 Dec 2002 10:52:26 -0600
Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS
Message-ID: <15852.57674.16727.930806@montanaro.dyndns.org>

Do we really need both Doc/whatsnew/* and Misc/NEWS?  It seems that people
frequently make nearly the same edits to both files, or add to Misc/NEWS
only and someone later (usually Andrew) goes back and twiddles the necessary
bits in Doc/whatsnew.

Skip



From just@letterror.com  Tue Dec  3 17:10:09 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 18:10:09 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E4A@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-184595BA06E211D7ADDD003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> Perhaps you could refactor your implementation so that the
> cache isn't just for zipfiles? (I can easily imagine other
> extensions that would benefit - tarfiles, metakit databases,
> imports over the web, ...)

The directory listing is the hardest part and can't be general; caching is as
simple as using a dict. I think it's the responsibility of the import hook.

Btw. I could provide a zip import hook that's not specific to one archive, but
rather searches sys.path for zip files. I'll play with that.

Just


From mwh@python.net  Tue Dec  3 17:30:28 2002
From: mwh@python.net (Michael Hudson)
Date: 03 Dec 2002 17:30:28 +0000
Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS
In-Reply-To: Skip Montanaro's message of "Tue, 3 Dec 2002 10:52:26 -0600"
References: <15852.57674.16727.930806@montanaro.dyndns.org>
Message-ID: <2m7ker57t7.fsf@starship.python.net>

Skip Montanaro <skip@pobox.com> writes:

> Do we really need both Doc/whatsnew/* and Misc/NEWS?  It seems that people
> frequently make nearly the same edits to both files, or add to Misc/NEWS
> only and someone later (usually Andrew) goes back and twiddles the necessary
> bits in Doc/whatsnew.

I'd have said yes: Misc/NEWS is for "developers" and whatsnew is for
"users", although this is blatantly a bogus categorization.

NEWS items are brief to the point of curtness, whatsnew has more
explanation and some examples.  Also I'd hope/expect NEWS to be more
comprehensive -- whatsnew shouldn't need to have every little thing in
it.

Cheers,
M.

-- 
  at any rate, I'm satisfied that not only do they know which end of
  the pointy thing to hold, but where to poke it for maximum effect.
                                  -- Eric The Read, asr, on google.com


From amk@amk.ca  Tue Dec  3 17:23:08 2002
From: amk@amk.ca (A.M. Kuchling)
Date: Tue, 03 Dec 2002 12:23:08 -0500
Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS
In-Reply-To: <15852.57674.16727.930806@montanaro.dyndns.org>
References: <15852.57674.16727.930806@montanaro.dyndns.org>
Message-ID: <asip5f$9uf$1@main.gmane.org>

Skip Montanaro wrote:
> Do we really need both Doc/whatsnew/* and Misc/NEWS?  

Yes, we do.  Misc/NEWS is complete, listing everything that's changed 
since the previous version. whatsnew is friendly, and there's a
cutoff point; some changes are too trivial or esoteric to be worth 
mentioning.

Feel free to only edit Misc/NEWS; before releases I'll go through it
to find things I've missed, so there's no need to update whatsnew23.tex.
I don't mind if people edit whatsnew23.tex to add items, but it's 
certainly not necessary.

--amk






From martin@v.loewis.de  Tue Dec  3 17:42:19 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 18:42:19 +0100
Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS
In-Reply-To: <2m7ker57t7.fsf@starship.python.net>
References: <15852.57674.16727.930806@montanaro.dyndns.org>
 <2m7ker57t7.fsf@starship.python.net>
Message-ID: <m3lm37ov7o.fsf@mira.informatik.hu-berlin.de>

Michael Hudson <mwh@python.net> writes:

> I'd have said yes: Misc/NEWS is for "developers" and whatsnew is for
> "users", although this is blatantly a bogus categorization.
> 
> NEWS items are brief to the point of curtness, whatsnew has more
> explanation and some examples.  Also I'd hope/expect NEWS to be more
> comprehensive -- whatsnew shouldn't need to have every little thing in
> it.

+1. NEWS ought to be complete, with respect to changes that could
remotely effect existing code. whatsnew.tex should focus on things
that are both new and useful.

Regards,
Martin



From martin@v.loewis.de  Tue Dec  3 17:47:59 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 18:47:59 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-DFEE7F9606CE11D7ADDD003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-DFEE7F9606CE11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <m3hedvouy8.fsf@mira.informatik.hu-berlin.de>

Just van Rossum <just@letterror.com> writes:

> I'm working hard on a zipimport module that uses
> sys.import_hooks. It's going to be SO much cleaner than the existing
> zip import patch.

Ok. I then won't apply the existing patch until
a) your zipfile patch comes along to review, or
b) significant time has passed after 2.3a1 (for some reasonable value
   for "significant")

IOW, it would be fine for me if 2.3a1 does not have zipfile import,
but I think it must be distributed with 2.3 final (and the betas
before it).

Regards,
Martin


From mwh@python.net  Tue Dec  3 18:02:12 2002
From: mwh@python.net (Michael Hudson)
Date: 03 Dec 2002 18:02:12 +0000
Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1
In-Reply-To: tim_one@users.sourceforge.net's message of "Tue, 03 Dec 2002 09:50:12 -0800"
References: <E18JHBU-0000Rd-00@sc8-pr-cvs1.sourceforge.net>
Message-ID: <2misybq8uz.fsf@starship.python.net>

tim_one@users.sourceforge.net writes:

> New program just to display pickle sizes.  This makes clear that the
> copy_reg based C implementation is much more space-efficient in the
> end than the __getstate__/__setstate__ based Python implementation,
> but that 4-byte date objects still suffer > 10 bytes of overhead each
> no matter how many of them you pickle in one gulp.

Presumably there's a possibility of an optimization for pickling
homogeneous (i.e. all the same type) lists (in pickle.py, not here).

Hard to say whether it would be worth it, though.

Cheers,
M.

-- 
  This is the fixed point problem again; since all some implementors
  do is implement the compiler and libraries for compiler writing, the
  language becomes good at writing compilers and not much else!
                                 -- Brian Rogoff, comp.lang.functional


From mwh@python.net  Tue Dec  3 18:27:59 2002
From: mwh@python.net (Michael Hudson)
Date: 03 Dec 2002 18:27:59 +0000
Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1
In-Reply-To: Michael Hudson's message of "03 Dec 2002 18:02:12 +0000"
References: <E18JHBU-0000Rd-00@sc8-pr-cvs1.sourceforge.net> <2misybq8uz.fsf@starship.python.net>
Message-ID: <2mfztfq7o0.fsf@starship.python.net>

Michael Hudson <mwh@python.net> writes:

> tim_one@users.sourceforge.net writes:
> 
> > New program just to display pickle sizes.  This makes clear that the
> > copy_reg based C implementation is much more space-efficient in the
> > end than the __getstate__/__setstate__ based Python implementation,
> > but that 4-byte date objects still suffer > 10 bytes of overhead each
> > no matter how many of them you pickle in one gulp.
> 
> Presumably there's a possibility of an optimization for pickling
> homogeneous (i.e. all the same type) lists (in pickle.py, not here).
> 
> Hard to say whether it would be worth it, though.

Here's a fairly simple minded patch to the pickling side of pickle.py:
it seems to save about 6 bytes per object in the good cases.

with:
list of 100 dates via      C -- 1236 bytes, 12.36 bytes/obj

without:
list of 100 dates via      C -- 1871 bytes, 18.71 bytes/obj

I'm not going to pursue this further unless someone thinks it's a
worthwhile move.

Cheers,
M.

Index: pickle.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/pickle.py,v
retrieving revision 1.72
diff -c -r1.72 pickle.py
*** pickle.py   13 Nov 2002 22:01:26 -0000      1.72
--- pickle.py   3 Dec 2002 18:24:37 -0000
***************
*** 109,114 ****
--- 109,115 ----
  INST            = 'i'
  LONG_BINGET     = 'j'
  LIST            = 'l'
+ HOM_LIST        = 'k'
  EMPTY_LIST      = ']'
  OBJ             = 'o'
  PUT             = 'p'
***************
*** 439,445 ****
--- 440,478 ----
      def save_empty_tuple(self, object):
          self.write(EMPTY_TUPLE)
  
+     def save_hom_list(self, object):
+         reduce = dispatch_table[type(object[0])]
+         
+         write = self.write
+         save  = self.save
+         memo  = self.memo
+ 
+         write(HOM_LIST)
+         
+         o = object[0]
+         
+         c, a = reduce(o)
+ 
+         l = [a]
+         
+         for o in object[1:]:
+             l.append(reduce(o)[1])
+ 
+         save(c)
+         save(l)
+ 
      def save_list(self, object):
+         t = {}
+ 
+         for o in object:
+             t[type(o)] = 1
+             if len(t) > 1:
+                 break
+         else:
+             if t and dispatch_table.has_key(t.iterkeys().next()):
+                 self.save_hom_list(object)
+                 return
+ 
          d = id(object)
  
          write = self.write

-- 
  Unfortunately, nigh the whole world is now duped into thinking that 
  silly fill-in forms on web pages is the way to do user interfaces.  
                                        -- Erik Naggum, comp.lang.lisp


From martin@v.loewis.de  Tue Dec  3 18:05:40 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 19:05:40 +0100
Subject: [Python-Dev] Make universal newlines non-optional
Message-ID: <m33cpfou4r.fsf@mira.informatik.hu-berlin.de>

At the time the universal newline code was committed, there was
discussion whether or not it should be optional.

I'd like to propose that it doesn't need to be optional: having a
configuration option for it complicates the build documentation, and
the space savings that one may gain from disabling it are minimal.

So I volunteer to remove the #ifdef machinery around it. Any
objections?

Regards,
Martin



From tim.one@comcast.net  Tue Dec  3 18:52:55 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 03 Dec 2002 13:52:55 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime
 picklesize.py,NONE,1.1
In-Reply-To: <2misybq8uz.fsf@starship.python.net>
Message-ID: <BIEJKCLHCIOIHAGOKOLHAEJBEBAA.tim.one@comcast.net>

[Tim]
>> New program just to display pickle sizes.  This makes clear that the
>> copy_reg based C implementation is much more space-efficient in the
>> end than the __getstate__/__setstate__ based Python implementation,
>> but that 4-byte date objects still suffer > 10 bytes of overhead each
>> no matter how many of them you pickle in one gulp.

[Michael Hudson]
> Presumably there's a possibility of an optimization for pickling
> homogeneous (i.e. all the same type) lists (in pickle.py, not here).
>
> Hard to say whether it would be worth it, though.

I don't really care about lists of date objects.  The intent was just to see
how much of the administrative pickle bloat could be saved by pickle's
internal memo facility when multiple date objects appear in a structure for
*whatever* reason (be it a list or tuple of dates, or a dict keyed by dates,
or an object with multiple data-value attributes, or ...).

The administrative pickle bloat for a single date instance is severe:

    pickling 2000-12-13 via Python -- pickle length 80
    pickling 2000-12-13 via C -- pickle length 43

The internal pickle memo saves a lot if more than one data instance appears,
via "remembering" parts of the overhead scaffolding.  But in the end a data
object has a 4-byte state string, and there's still a lot more overhead than
state stored in the pickles.

> ...
> Here's a fairly simple minded patch to the pickling side of pickle.py:
> it seems to save about 6 bytes per object in the good cases.
>
> with:
> list of 100 dates via      C -- 1236 bytes, 12.36 bytes/obj
>
> without:
> list of 100 dates via      C -- 1871 bytes, 18.71 bytes/obj

The "without" number is most curious.  When I run the checked-in code, I get

  list of 100 dates via      C -- 1533 bytes, 15.33 bytes/obj

This was w/ CVS Python, Win2K and MSVC6.  Ah!  You must have fiddled the
script to use pickle instead of cPickle.  I get 1871 bytes then.  I'm
surprised they're so different.

> I'm not going to pursue this further unless someone thinks it's a
> worthwhile move.

I don't personally have large pickles of homogeneous lists, so hard to say.
The C implementation of pickle would also need to be fiddled.  Cutting the
pickle overheads for instances of new-style classes would more clearly be
worthwhile.



From bac@OCF.Berkeley.EDU  Tue Dec  3 18:56:13 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Tue, 3 Dec 2002 10:56:13 -0800 (PST)
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>
 <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>
 <0ad001c29abf$ed602f50$0900a8c0@spiff>
 <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>

[Guido van Rossum]

> > > >     try:
> > > >         import threading
> > > >     except ImportError:
> > > >         threading = None # use non-threaded behaviour
> > >
> > > But then you'd have to sprinkle the rest of the code with
> > > "if threading is not None" tests.  Brett's dummy_thread is an attempt
> > > to reduce this clutter.
> >
> > the problem (if there is one) is that the proposed integration may
> > break code that uses the above pattern.
>

But if it does it will be on platforms it couldn't run on previously.
``threading`` uses ``thread`` when possible and only falls back on
``dummy_thread`` when it isn't available.  That means this could only be a
problem where  the code  was not even runnable before.

But yes, there are chances of deadlock in certain situations.  It
basically comes down to whether we want to protect users from themselves
by not having them get into a possible deadlock situation.

> I don't propose to automate this.  I propose this:
>
>   try:
>     import threading
>   except ImportError:
>     import dummy_threading as threading
>

So is this an indirect request for me to write ``dumy_threading``?  If it
is it won't be hard: I would do an ``import *`` on ``threading`` and then
override the functions it uses from ``thread`` with ``import ... from ...
as ...`` from ``dummy_thread``.  Also would mean rewriting the
``threading`` testing suite.  Prefectly happy to write it, just need you
to say you want it.

-Brett


From fredrik@pythonware.com  Tue Dec  3 19:24:31 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Tue, 3 Dec 2002 20:24:31 +0100
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU> <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>              <0ad001c29abf$ed602f50$0900a8c0@spiff>  <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
Message-ID: <005b01c29b01$9ca50c10$ced241d5@hagrid>

Brett Cannon wrote:
> 
> > > > >     try:
> > > > >         import threading
> > > > >     except ImportError:
> > > > >         threading = None # use non-threaded behaviour

> But if it does it will be on platforms it couldn't run on previously.

Sigh.  Let's try one more time:

Under 2.2 and earlier, on a platform that doesn't support threads,
the above code will set threading to None, and the application can
then use "if threading is None" to provide application-specific work-
arounds.

(catching import errors is a very common pattern when you're writing
portable Python code).

If you change Python so that "import threading" succeeds also on plat-
forms that don't support threads, the application-specific workarounds
will no longer be used, and the application may no longer work properly.

And since we're talking deadlocks, chances are that nobody will notice
before it's too late.

</F>



From guido@python.org  Tue Dec  3 19:29:43 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 14:29:43 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Tue, 03 Dec 2002 17:05:24 +0100."
 <016e01c29ae5$c9ec73a0$6d94fea9@newmexico>
References: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com>
 <016e01c29ae5$c9ec73a0$6d94fea9@newmexico>
Message-ID: <200212031929.gB3JThr04042@pcp02138704pcs.reston01.va.comcast.net>

> >1. Your import hooks seem to act at a level "above" sys.path,
> >   in the sense that the normal sys.path search becomes just
> >   another hook. With that in mind, I don't see a way for a user
> >   to add a zipfile in the *middle* of sys.path.
> 
> It is worth to notice that Jython 2.1 already implement more or less
> PEP 273 approach, allowing to add zipfiles to sys.path.

I'd be disappointed if adding zipfiles to sys.path wasn't possible.
After all that's what PEP 273 says.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Tue Dec  3 19:34:28 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 14:34:28 -0500
Subject: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS
In-Reply-To: Your message of "Tue, 03 Dec 2002 10:52:26 CST."
 <15852.57674.16727.930806@montanaro.dyndns.org>
References: <15852.57674.16727.930806@montanaro.dyndns.org>
Message-ID: <200212031934.gB3JYT004084@pcp02138704pcs.reston01.va.comcast.net>

> Do we really need both Doc/whatsnew/* and Misc/NEWS?  It seems that people
> frequently make nearly the same edits to both files, or add to Misc/NEWS
> only and someone later (usually Andrew) goes back and twiddles the necessary
> bits in Doc/whatsnew.

They serve different purposes.  Misc/NEWS is supposed to be
exhaustive, but terse.  whatsnew23.tex is supposed to give the
highlights, with examples, etc.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From bac@OCF.Berkeley.EDU  Tue Dec  3 19:36:44 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Tue, 3 Dec 2002 11:36:44 -0800 (PST)
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <005b01c29b01$9ca50c10$ced241d5@hagrid>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>
 <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>
 <0ad001c29abf$ed602f50$0900a8c0@spiff>
 <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
 <005b01c29b01$9ca50c10$ced241d5@hagrid>
Message-ID: <Pine.SOL.4.50.0212031130080.29291-100000@death.OCF.Berkeley.EDU>

[Fredrik Lundh]

> Brett Cannon wrote:
> >
> > > > > >     try:
> > > > > >         import threading
> > > > > >     except ImportError:
> > > > > >         threading = None # use non-threaded behaviour
>
> > But if it does it will be on platforms it couldn't run on previously.
>
> Sigh.  Let's try one more time:
>
> Under 2.2 and earlier, on a platform that doesn't support threads,
> the above code will set threading to None, and the application can
> then use "if threading is None" to provide application-specific work-
> arounds.
>

I get this and always have.  I just have not thought of it as huge of an
issue as you  do.  If  it  is that big of a worry, add a ``import thread;
del thread`` statement to make sure  ``thread`` is there and warn users
not to use the code.

But I know you are going to be against the idea of modifying existing code
for this, so I am going to let it go.  Your point is valid, we just differ
on its importance.

Is Guido's suggested ``dummy_threading`` okay with you?

> (catching import errors is a very common pattern when you're writing
> portable Python code).
>

I know, I use it myself.

> If you change Python so that "import threading" succeeds also on plat-
> forms that don't support threads, the application-specific workarounds
> will no longer be used, and the application may no longer work properly.
>
> And since we're talking deadlocks, chances are that nobody will notice
> before it's too late.
>

OK, I am going to concede on this and stop pushing for it.  Let me know
what you think of Guido's ``dummy_threading`` idea.

I assume you don't object to changing ``Queue``, though, right?

-Brett


From pedronis@bluewin.ch  Tue Dec  3 19:28:41 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 20:28:41 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <16E1010E4581B049ABC51D4975CEDB8861996A@UKDCX001.uk.int.atosorigin.com>              <016e01c29ae5$c9ec73a0$6d94fea9@newmexico>  <200212031929.gB3JThr04042@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <003d01c29b02$30471580$6d94fea9@newmexico>

From: "Guido van Rossum" <guido@python.org>
> > >1. Your import hooks seem to act at a level "above" sys.path,
> > >   in the sense that the normal sys.path search becomes just
> > >   another hook. With that in mind, I don't see a way for a user
> > >   to add a zipfile in the *middle* of sys.path.
> >
> > It is worth to notice that Jython 2.1 already implement more or less
> > PEP 273 approach, allowing to add zipfiles to sys.path.
>
> I'd be disappointed if adding zipfiles to sys.path wasn't possible.
> After all that's what PEP 273 says.

but - if I understand correctly - given the approach taken by the import_hooks
patch
there's the issue:

consider all directories then all zipfiles

versus consider both kind of import sources according to the order in sys.path,
which is the most natural extension.

regards.




From martin@v.loewis.de  Tue Dec  3 19:55:39 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 20:55:39 +0100
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>
 <077101c29a13$a9b78cf0$0900a8c0@spiff>
 <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>
 <0ad001c29abf$ed602f50$0900a8c0@spiff>
 <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
Message-ID: <m3ptsiop1g.fsf@mira.informatik.hu-berlin.de>

Brett Cannon <bac@OCF.Berkeley.EDU> writes:

> > I don't propose to automate this.  I propose this:
> >
> >   try:
> >     import threading
> >   except ImportError:
> >     import dummy_threading as threading
> >
> 
> So is this an indirect request for me to write ``dumy_threading``?  

I would suggest to make this 

    import dummy_thread as threading

i.e. have dummy_thread expose the union of the thread and threading
APIs.

> If it is it won't be hard: I would do an ``import *`` on
> ``threading`` 

That wont't work: "from threading import *" will raise an ImportError.

I suggest to add dummy_thread to sys.modules, then execfile
threading.py in dummy_thread, then remove it from sys.modules.

Regards,
Martin


From guido@python.org  Tue Dec  3 19:58:31 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 14:58:31 -0500
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: Your message of "Tue, 03 Dec 2002 10:56:13 PST."
 <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU> <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
Message-ID: <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net>

> > I don't propose to automate this.  I propose this:
> >
> >   try:
> >     import threading
> >   except ImportError:
> >     import dummy_threading as threading
> >
> 
> So is this an indirect request for me to write ``dumy_threading``?  If it
> is it won't be hard: I would do an ``import *`` on ``threading`` and then
> override the functions it uses from ``thread`` with ``import ... from ...
> as ...`` from ``dummy_thread``.  Also would mean rewriting the
> ``threading`` testing suite.  Prefectly happy to write it, just need you
> to say you want it.

Yes.  I don't want threading to automatically substitute dummy_thread
for thread.  Multiple imports of threading must all fail when thread
cannot be imported.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Tue Dec  3 20:05:52 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 21:05:52 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <003d01c29b02$30471580$6d94fea9@newmexico>
Message-ID: <r01050400-1022-A464CECE06FA11D7ADDD003065D5E7E4@[10.0.0.23]>

[Guido]
> > I'd be disappointed if adding zipfiles to sys.path wasn't possible.
> > After all that's what PEP 273 says.

[Samuele Pedroni]
> but - if I understand correctly - given the approach taken by the
> import_hooks patch there's the issue:
> 
> consider all directories then all zipfiles

(or vice versa ;-)

> versus consider both kind of import sources according to the order in
> sys.path, which is the most natural extension.

Right. I admit I dismissed sys.path a wee bit too quickly when proposing an
alternative zipimport mechanism.

Regarding sys.import_hooks, I see three possibilities:

#1
    for p in sys.path:
        for hook in sys.import_hooks:
            hook(p, ...)

#2
    for hook in sys.import_hooks:
        for p in sys.path:
            hook(p, ...)

#3
    for hook in sys.import_hooks:
        # the hook can traversing sys.path if needed.
        hook(...)

My current sys.import_hooks patch implements #3 and I think that's the most
logical thing to do; after all sys.path may not be meaningful for a hook. #2
makes little sense I guess. #1 could be an option, but, given the current state
of import.c, would be a lot harder to implement. I also think that #3 is the
most efficient scheme, as the hook can save some local state while traversing
sys.path.

I will change my zipimport hook (which _still_ isn't finished ;-) to also
traverse sys.path, and check for *.zip path endings.

Just


From pedronis@bluewin.ch  Tue Dec  3 20:03:13 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 21:03:13 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-A464CECE06FA11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <007301c29b07$02de61c0$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> I will change my zipimport hook (which _still_ isn't finished ;-) to also
> traverse sys.path, and check for *.zip path endings.
>

my point was that, given

dir1;zip0;dir2

dir1 should take over zip0 that should take over dir2.

[This is what Jython 2.1 does btw]

If the resulting import precedence order is that both dir1 and dir2 take over
zip0, then maybe allowing zip0 in sys.path is just confusing,
and zipfiles should be specified elsewhere.

In general I clearly see that there can be import hooks that can leverage
sys.path and its order semantics and some that can't.

regards.



From guido@python.org  Tue Dec  3 20:09:58 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 03 Dec 2002 15:09:58 -0500
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: Your message of "Tue, 03 Dec 2002 11:36:44 PST."
 <Pine.SOL.4.50.0212031130080.29291-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU> <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU> <005b01c29b01$9ca50c10$ced241d5@hagrid>
 <Pine.SOL.4.50.0212031130080.29291-100000@death.OCF.Berkeley.EDU>
Message-ID: <200212032009.gB3K9wP04418@pcp02138704pcs.reston01.va.comcast.net>

> > Under 2.2 and earlier, on a platform that doesn't support threads,
> > the above code will set threading to None, and the application can
> > then use "if threading is None" to provide application-specific work-
> > arounds.
> 
> I get this and always have.  I just have not thought of it as huge of an
> issue as you  do.

It is a big deal, Brett.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Tue Dec  3 20:45:35 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 21:45:35 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <007301c29b07$02de61c0$6d94fea9@newmexico>
Message-ID: <r01050400-1022-31B1BA8A070011D7ADDD003065D5E7E4@[10.0.0.23]>

Samuele Pedroni wrote:

> From: "Just van Rossum" <just@letterror.com>
> > I will change my zipimport hook (which _still_ isn't finished ;-) to also
> > traverse sys.path, and check for *.zip path endings.
> >
> 
> my point was that, given
> 
> dir1;zip0;dir2
> 
> dir1 should take over zip0 that should take over dir2.
> 
> [This is what Jython 2.1 does btw]
> 
> If the resulting import precedence order is that both dir1 and dir2 take over
> zip0, then maybe allowing zip0 in sys.path is just confusing,
> and zipfiles should be specified elsewhere.

True, true.

Does Jython implement the (old) idea that items on sys.path can be import
handler *objects*? (If yes, can you briefly describe the interface of these
objects?)

After seeing what you mean I think that may be the only sensible solution. 

Say:

    for p in sys.path:
        if hasattr(p, "find_module"):
            x = p.find_module(...)
        elif not isinstance(p, basestring):
            continue
        else:
            x = builtin_find_module(...)
        ...load module...

Just


From martin@v.loewis.de  Tue Dec  3 20:49:28 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 03 Dec 2002 21:49:28 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-31B1BA8A070011D7ADDD003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-31B1BA8A070011D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <m34r9uomjr.fsf@mira.informatik.hu-berlin.de>

Just van Rossum <just@letterror.com> writes:

> Does Jython implement the (old) idea that items on sys.path can be import
> handler *objects*? (If yes, can you briefly describe the interface of these
> objects?)
> 
> After seeing what you mean I think that may be the only sensible solution. 

I think one requirement is that you can put zipfiles in PYTHONPATH and
have that work.

Regards,
Martin


From pedronis@bluewin.ch  Tue Dec  3 20:43:11 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 21:43:11 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-31B1BA8A070011D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <00f501c29b0c$989af480$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>

> Samuele Pedroni wrote:
>
> > From: "Just van Rossum" <just@letterror.com>
> > > I will change my zipimport hook (which _still_ isn't finished ;-) to also
> > > traverse sys.path, and check for *.zip path endings.
> > >
> >
> > my point was that, given
> >
> > dir1;zip0;dir2
> >
> > dir1 should take over zip0 that should take over dir2.
> >
> > [This is what Jython 2.1 does btw]
> >
> > If the resulting import precedence order is that both dir1 and dir2 take
over
> > zip0, then maybe allowing zip0 in sys.path is just confusing,
> > and zipfiles should be specified elsewhere.
>
> True, true.
>
> Does Jython implement the (old) idea that items on sys.path can be import
> handler *objects*? (If yes, can you briefly describe the interface of these
> objects?)
>

No jython accepts strings of the form *.zip in sys.path, that are converted
upon scanning on import  to string subclasses' instances with the same value as
string, through which import happens using an internal interface implemented by
these string subclasses.

We also set __path__s to contain things of the form *.zip/rel/path/to/package.

regards.



From mchermside@ingdirect.com  Tue Dec  3 21:11:35 2002
From: mchermside@ingdirect.com (Chermside, Michael)
Date: Tue, 3 Dec 2002 16:11:35 -0500
Subject: [Python-Dev] Re: New and Improved Import Hooks
Message-ID: <902A1E710FEAB740966EC991C3A38A8903C35B21@INGDEXCHANGEC1.ingdirect.com>

> > 1. Your import hooks seem to act at a level "above" sys.path,
> >    in the sense that the normal sys.path search becomes just
> >    another hook. With that in mind, I don't see a way for a user
> >    to add a zipfile in the *middle* of sys.path.
>=20
> True. There would be ways around that, but it's currently not easy. I =
don't feel
> this is a particularly important feature, though.

Hmm... Not sure I agree. I think this is one area where we'd
be wise to follow Java's example. Normally I think it's wise
for Python to do its own design instead of imitating the errors
of other languages, but this is the exception.

  (1) Java (AFAIK) originated the idea of mixing .zip files as
      part of the path.
  (2) Their approach makes sense (in my mind, it would make sense
      to say "zip archives appear in path and are treated as if they=20
      were subdirectories", and it would ALSO make sense to say
      "the list of zip archives is separate from the path". But some
      mix like "zip archives appear in the path, but we treat them
      as if they were at the end even though they aren't" doesn't
      make sense).
  (3) It makes sense for Jython (for obvious reasons!).

Because of (1), along with the popularity of java, putting .zip's
on the path will seem "natural and familiar" to lots of Python
programmers. Because of (2) it isn't a BAD idea. And (3) is just
another good motivation... let's keep c-python and jython as
similar as possible!

I sympathize about this being "not currently easy", but...

-- Michael Chermside

PS: I like the import-hooks idea, and I think the zip importer is
an excellent test case to really see how well the concept plays
in practice.


From just@letterror.com  Tue Dec  3 21:20:57 2002
From: just@letterror.com (Just van Rossum)
Date: Tue,  3 Dec 2002 22:20:57 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <00f501c29b0c$989af480$6d94fea9@newmexico>
Message-ID: <r01050400-1022-204AF340070511D7ADDD003065D5E7E4@[10.0.0.23]>

Samuele Pedroni wrote:

> No jython accepts strings of the form *.zip in sys.path, that are
> converted upon scanning on import  to string subclasses' instances
> with the same value as string, through which import happens using an
> internal interface implemented by these string subclasses.
> 
> We also set __path__s to contain things of the form *.zip/rel/path/to/package.

And these package __path__s are _also_ "import hook" string subclasses, right?
That makes an awful lot of sense.

Just


From pedronis@bluewin.ch  Tue Dec  3 21:44:48 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 3 Dec 2002 22:44:48 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-204AF340070511D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <01bf01c29b15$34474660$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> Samuele Pedroni wrote:
>
> > No jython accepts strings of the form *.zip in sys.path, that are
> > converted upon scanning on import  to string subclasses' instances
> > with the same value as string, through which import happens using an
> > internal interface implemented by these string subclasses.
> >
> > We also set __path__s to contain things of the form
*.zip/rel/path/to/package.
>
> And these package __path__s are _also_ "import hook" string subclasses,
right?

Yes

The interface is internal because here was no CPython prior art, and because we
also support importing java classes from sys.path (both from dirs and zipfiles,
although there are some bugs wrt to the latter), and our import code also begs
for some refactoring... so things are in flux.

[In the future (2.2) we were planning to add support for Java ClassLoaders too,
but it is still unclear whether to put them in sys.path makes sense or not. I'm
reporting this not because Java ClassLoaders are relevant to CPython but to
point out that probably not all import hooks can be sys.path centered, and then
a question is whether they should be considered before or after sys.path ]

regards.



From tdelaney@avaya.com  Tue Dec  3 23:25:18 2002
From: tdelaney@avaya.com (Delaney, Timothy)
Date: Wed, 4 Dec 2002 10:25:18 +1100
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <B43D149A9AB2D411971300B0D03D7E8BF0A7AC@natasha.auslabs.avaya.com>

> From: Just van Rossum [mailto:just@letterror.com]
> 
> Regarding sys.import_hooks, I see three possibilities:
> 
> #1
>     for p in sys.path:
>         for hook in sys.import_hooks:
>             hook(p, ...)
> 
> #2
>     for hook in sys.import_hooks:
>         for p in sys.path:
>             hook(p, ...)
> 
> #3
>     for hook in sys.import_hooks:
>         # the hook can traversing sys.path if needed.
>         hook(...)

It seems to me there needs to be two separate sets of import hooks.

1. Module import hooks.
2. Source import hooks.

Module import hooks are the type being discussed for sys.import_hooks - a
sequence of callables which are called one after the other until a module is
successfully imported.

Source import hooks provide the exact mechanism for doing an import from a
particular data source.

Doing this, in effect we would have the following algorithm ...

    for hook in sys.import_hooks:
        hook(...)

and the algorithm for the builtin import hook would be ...

    for p in sys.path:
        for hook in sys.source_import_hooks:
            hook(p, ...)

which would end up being the default behaviour.

Individual module import hooks could (and probably should) use the
appropriate source import hooks.

I think this would suit all requirements, whilst being both general and
clean. It would provide a simple mechanism for allowing .tar files in
sys.path, internet imports, etc.

Tim Delaney


From bac@OCF.Berkeley.EDU  Wed Dec  4 01:49:12 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Tue, 3 Dec 2002 17:49:12 -0800 (PST)
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>
 <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>
 <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
 <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <Pine.SOL.4.50.0212031747240.27-100000@death.OCF.Berkeley.EDU>

[Guido van Rossum]

> > > I don't propose to automate this.  I propose this:
> > >
> > >   try:
> > >     import threading
> > >   except ImportError:
> > >     import dummy_threading as threading
> > >
> >
> > So is this an indirect request for me to write ``dumy_threading``?  If it
> > is it won't be hard: I would do an ``import *`` on ``threading`` and then
> > override the functions it uses from ``thread`` with ``import ... from ...
> > as ...`` from ``dummy_thread``.  Also would mean rewriting the
> > ``threading`` testing suite.  Prefectly happy to write it, just need you
> > to say you want it.
>
> Yes.  I don't want threading to automatically substitute dummy_thread
> for thread.  Multiple imports of threading must all fail when thread
> cannot be imported.
>

Do you prefer I do a separate ``dummy_threading`` module or follow
Martin's suggestions and just turn ``dummy_thread`` into a ``threading``
substitute itself?

-Brett


From shane.holloway@ieee.org  Wed Dec  4 02:28:51 2002
From: shane.holloway@ieee.org (Shane Holloway (IEEE))
Date: Tue, 03 Dec 2002 19:28:51 -0700
Subject: [Python-Dev] Re: New and Improved Import Hooks
Message-ID: <3DED6863.6000100@ieee.org>

    As it seems I am supposed to introduce myself on my first post, I am 
a professional software engineer in Colorado, USA, blessed with the 
chance to implement my solutions in Python.  I've a fairly diverse 
background in languages, but the most experience with C++.  I keep 
working on the RuneBlade project <http://www.runeblade.com>, which helps 
me deliver my Python solutions just that much faster.  Seaching for a 
better language, Python has satisfied my thirst, seeming to combine my 
best experiences from Lisp, Pascal, C++ and others.  Thanks for all of 
your hard work!  Now it's time for me to try and help!

    IMO, Just's approach seems to be flexable enough to solve more than 
the problem at hand.  Not only does the framework help to implement PEP 
273, allowing for import from zip files, but it provides a very logical 
place for other import mechanisms to be implemented.  Expanding upon Tim 
Delaney's desires to include importing from tar files and internet 
references, consider being able to extend this mechanism to importing 
from an XML document, database of modules, bz2 files, and even encrypted 
collections,  The key, as I see it, is the flexability of the approach.  
One can even add and remove import stratigies on the fly without prior 
knowledge of all the import hooks!  :-)

    As for keeping the import order of the sys.path, Tim seems to have 
hit the mark with the default importer supporting "sub-hooks" for the 
different path entry types.  The best part about the import_hooks 
approach is that it leaves enough flexability with developer to 
implement creative approaches, while maintaining a commonly recognized 
standard for import hook interfaces.  Even module-caching import hooks 
can use this model.  One could write a simple hook that would log all 
imports, or all imports that could not be found, regardless of others 
import hooks.  (Well, at least, if they behave properly ;)

-Shane Holloway




From tdelaney@avaya.com  Wed Dec  4 02:36:52 2002
From: tdelaney@avaya.com (Delaney, Timothy)
Date: Wed, 4 Dec 2002 13:36:52 +1100
Subject: [Python-Dev] Re: New and Improved Import Hooks
Message-ID: <B43D149A9AB2D411971300B0D03D7E8BF0A7AD@natasha.auslabs.avaya.com>

> From: Shane Holloway (IEEE) [mailto:shane.holloway@ieee.org]
>
> Expanding upon Tim 
> Delaney's desires to include importing from tar files and internet 
> references,

Hey - I didn't say *I* wanted this (they came from other posts ;) but it's
an obvious extension ...

> consider being able to extend this mechanism to importing 
> from an XML document, database of modules, bz2 files, and 
> even encrypted 
> collections,

As are all of these.

I think if the import mechanism is going to be changed, it should be changed
so as to be flexible. The fact that such a change is also (IMO) a much
cleaner approach is icing.

Tim Delaney


From tim.one@comcast.net  Wed Dec  4 03:12:29 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 03 Dec 2002 22:12:29 -0500
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <Pine.SOL.4.50.0212031747240.27-100000@death.OCF.Berkeley.EDU>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEPMDAAB.tim.one@comcast.net>

[Guido]
>>>> I don't propose to automate this.  I propose this:
>>>>
>>>>   try:
>>>>       import threading
>>>>   except ImportError:
>>>>       import dummy_threading as threading

...

[Brett Cannon]
> Do you prefer I do a separate ``dummy_threading`` module or follow
> Martin's suggestions and just turn ``dummy_thread`` into
> a ``threading`` substitute itself?

Looking at his code, I think it's clear he wants a separate

    dummy_threading.py

and nothing fancier than that.  The idiomatic use would be as shown, where a
user who knows what they're doing can import dummy_threading (if desired)
and *name* it threading locally themself (if desired).  No existing code
could break, except for users who already have a file named
dummy_threading.py on their PYTHONPATH somewhere after the std library.



From bac@OCF.Berkeley.EDU  Wed Dec  4 04:27:20 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Tue, 3 Dec 2002 20:27:20 -0800 (PST)
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEPMDAAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAEPMDAAB.tim.one@comcast.net>
Message-ID: <Pine.SOL.4.50.0212032021540.11880-100000@death.OCF.Berkeley.EDU>

[Tim Peters]

> [Guido]
> >>>> I don't propose to automate this.  I propose this:
> >>>>
> >>>>   try:
> >>>>       import threading
> >>>>   except ImportError:
> >>>>       import dummy_threading as threading
>
> ...
>
> [Brett Cannon]
> > Do you prefer I do a separate ``dummy_threading`` module or follow
> > Martin's suggestions and just turn ``dummy_thread`` into
> > a ``threading`` substitute itself?
>
> Looking at his code, I think it's clear he wants a separate
>
>     dummy_threading.py
>
> and nothing fancier than that.  The idiomatic use would be as shown, where a
> user who knows what they're doing can import dummy_threading (if desired)
> and *name* it threading locally themself (if desired).  No existing code
> could break, except for users who already have a file named
> dummy_threading.py on their PYTHONPATH somewhere after the std library.
>

OK.  Then I will just whip together the module (most likely following
Martin's correction of my previous from-the-cuff idea on how to do it) and
add it to the patch.

So if anyone was planning on reviewing the patch, please hold off.  I will
email the list again once I have the module and the patches regenerated
(although someone can go ahead and delete all the files in the patch; I
will just put all the files up anew since practically all of them will
change).

-Brett



From rjones@ekit-inc.com  Wed Dec  4 04:47:14 2002
From: rjones@ekit-inc.com (Richard Jones)
Date: Wed, 4 Dec 2002 15:47:14 +1100
Subject: [Python-Dev] PEP 301: Package Index and Metadata for Distutils
Message-ID: <200212041547.14551.rjones@ekit-inc.com>

I've implemented all I believe is necessary for the package index to be 
pronounced accepted or not. This includes the last few missing pieces:

- library reference patch against latest CVS
- distutils/dist.py patch against latest CVS
- use of distutils.version.LooseVersion sorting in the index database

Where to now? Is there more to do? Should I stop now because it's not worth 
continuing?


     Richard



From guido@python.org  Wed Dec  4 08:44:21 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 04 Dec 2002 03:44:21 -0500
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: Your message of "Tue, 03 Dec 2002 17:49:12 PST."
 <Pine.SOL.4.50.0212031747240.27-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU> <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net> <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net> <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU> <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212031747240.27-100000@death.OCF.Berkeley.EDU>
Message-ID: <200212040844.gB48iLx06213@pcp02138704pcs.reston01.va.comcast.net>

> Do you prefer I do a separate ``dummy_threading`` module or follow
> Martin's suggestions and just turn ``dummy_thread`` into a ``threading``
> substitute itself?

Two separate modules.  They have different APIs and it wouldn't be
wise to export a different API in the dummy case than in the real
case.  (I'm sure you can figure out why.)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Wed Dec  4 09:59:56 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 04 Dec 2002 10:59:56 +0100
Subject: [Python-Dev] PEP 301: Package Index and Metadata for Distutils
References: <200212041547.14551.rjones@ekit-inc.com>
Message-ID: <3DEDD21C.9030909@lemburg.com>

Richard Jones wrote:
> I've implemented all I believe is necessary for the package index to be 
> pronounced accepted or not. This includes the last few missing pieces:
> 
> - library reference patch against latest CVS
> - distutils/dist.py patch against latest CVS
> - use of distutils.version.LooseVersion sorting in the index database
> 
> Where to now? Is there more to do? Should I stop now because it's not worth 
> continuing?

I like the idea.

One nit: in order to make the setup.py script work for Python 1.5.2
and up, you should provide a way which let's the distutils
versions of those platforms ignore the classification keyword
argument for setup().

It should be fairly easy, I believe (untested, but you get the picture...):

# Add 'classifiers' support
if sys.version < '2.3:
    from distutils.dist import Distribution, translate_longopt
    Distribution.display_options.append(
          ('classifiers', None, "print the list of classifiers (needs Python 2.3)"))
    Distribution.display_option_names = map(lambda x: translate_longopt(x[0]),
                                            Distribution.display_options)

setup(...)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From Jack.Jansen@cwi.nl  Wed Dec  4 10:15:56 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Wed, 4 Dec 2002 11:15:56 +0100
Subject: [Python-Dev] Make universal newlines non-optional
In-Reply-To: <m33cpfou4r.fsf@mira.informatik.hu-berlin.de>
Message-ID: <6089D968-0771-11D7-93BC-0030655234CE@cwi.nl>

On Tuesday, Dec 3, 2002, at 19:05 Europe/Amsterdam, Martin v. L=F6wis=20
wrote:

> At the time the universal newline code was committed, there was
> discussion whether or not it should be optional.
>
> I'd like to propose that it doesn't need to be optional: having a
> configuration option for it complicates the build documentation, and
> the space savings that one may gain from disabling it are minimal.
>
> So I volunteer to remove the #ifdef machinery around it. Any
> objections?

Well, we don't know yet whether it will work on non-mainstream=20
platforms.
It'll probably be okay on BeOS or Amiga or such (and maybe the=20
maintainers
of those platforms track CVS anyway?), but I wouldn't vouch for Windows=20=

CE
or so.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>       =20
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma=20
Goldman -



From Jack.Jansen@cwi.nl  Wed Dec  4 10:24:38 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Wed, 4 Dec 2002 11:24:38 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-A464CECE06FA11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <978F6A8C-0772-11D7-93BC-0030655234CE@cwi.nl>

On Tuesday, Dec 3, 2002, at 21:05 Europe/Amsterdam, Just van Rossum 
wrote:
> Regarding sys.import_hooks, I see three possibilities:
>
> #1
>     for p in sys.path:
>         for hook in sys.import_hooks:
>             hook(p, ...)
>
> #2
>     for hook in sys.import_hooks:
>         for p in sys.path:
>             hook(p, ...)
>
> #3
>     for hook in sys.import_hooks:
>         # the hook can traversing sys.path if needed.
>         hook(...)

Note that there is "prior art" for case 1: the PYC resource importer 
for MacPython, which also
looks at files (in stead of directories) on sys.path.

It does have a bit of hairy code so it skips sys.path entries that 
don't apply to it
(directories, basically). This was a major speedup, so maybe this 
should be generalized
to something like

	for p in sys.path:
		for hook in sys.import_hooks:
			if not hook_can_handle.has_key((hook, p)):
				hook_can_handle[(hook, p)] = hook(p, None) # None requests 
"compatibility test"
			if hook_can_handle[(hook, p)]:
				hook(p, ....)
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From fredrik@pythonware.com  Wed Dec  4 10:33:05 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 4 Dec 2002 11:33:05 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <978F6A8C-0772-11D7-93BC-0030655234CE@cwi.nl>
Message-ID: <11a801c29b80$89bb2e20$0900a8c0@spiff>

Jack Jansen wrote:

> > Regarding sys.import_hooks, I see three possibilities:
> >
> > #1
> >     for p in sys.path:
> >         for hook in sys.import_hooks:
> >             hook(p, ...)
> >
> > #2
> >     for hook in sys.import_hooks:
> >         for p in sys.path:
> >             hook(p, ...)
> >
> > #3
> >     for hook in sys.import_hooks:
> >         # the hook can traversing sys.path if needed.
> >         hook(...)

what's wrong with:

#4
    for path in sys.path:
        if isinstance(path, types.StringTypes):
            do_standard_import(...)
        else:
            path.do_import(...)

after all, everyone knows how the path works, and everyone knows how
to manipulate sys.path inside a Python program.

(for PIL and other plug-in architectures, a "path.get_list_of_modules"
method would also be nice).

</F>



From martin@v.loewis.de  Wed Dec  4 10:41:01 2002
From: martin@v.loewis.de (Martin v. Löwis)
Date: Wed, 4 Dec 2002 11:41:01 +0100
Subject: [Python-Dev] Make universal newlines non-optional
References: <6089D968-0771-11D7-93BC-0030655234CE@cwi.nl>
Message-ID: <002b01c29b81$a4689900$df3fe8d9@mira>

> Well, we don't know yet whether it will work on non-mainstream
> platforms. It'll probably be okay on BeOS or Amiga or such
> (and maybe the  maintainers of those platforms track CVS
> anyway?), but I wouldn't vouch for Windows CE or so.

What would be potential consequences of it not working?

If it means that strange things may happen if you pass "U" to open, the
feature could be built, anyway.

Also, do you doubt that it *could* work (with sufficient modifications)
on all systems we care about? If so, users of these systems would need
to provide the necessary changes.

Regards,
Martin



From Jack.Jansen@cwi.nl  Wed Dec  4 11:02:07 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Wed, 4 Dec 2002 12:02:07 +0100
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: <m365uby0r3.fsf@mira.informatik.hu-berlin.de>
Message-ID: <D464FBC2-0777-11D7-93BC-0030655234CE@cwi.nl>

On Tuesday, Dec 3, 2002, at 09:17 Europe/Amsterdam, Martin v. L=F6wis=20
wrote:

> Greg Ewing <greg@cosc.canterbury.ac.nz> writes:
>
>> If this involves VPATH, I think I might know what's happening --
>> something similar happened to me recently.
>>
>> Have you by any chance previously done a non-VPATH build in the =
source
>> tree?
>
> I get the same problem when building in the source directory
> (/usr/src/python). VPATH is not set in the Makefile, and the
> compilation commands read like this
>
> gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I.=20
> -I/usr/src/python/./Include -I/usr/local/include=20
> -I/usr/src/python/Include -I/usr/src/python -c=20
> /usr/src/python/Modules/regexmodule.c -o=20
> /usr/src/python/Modules/regexmodule.o
>
> I believe this is triggered by the change
>
> ----------------------------
> revision 1.45
> date: 2001/08/05 22:31:19;  author: jackjansen;  state: Exp;  lines:=20=

> +63 -6
> Replace moddir and incdir by
> moddirlist and incdirlist, lists of source and include
> directories that are searched for modules.
> This is needed because the Mac modules and include files
> live in the Mac subtree.

I backed up to 1.44, but I still see the same problem. And it stands to=20=

reason: my find_module_file()
call was a replacement for os.path.join(moddir, filename). And with=20
moddir being an absolute path
this will also return an absolute path.

I am tempted to think that this problem has always existed, but has=20
gone unnoticed until now.
Now that I know about the problem it clears up some strange behaviour=20
I've seen in the past.
I would love to see this fixed, but I'm afraid I'm not the person to do=20=

it. Distutils
never ceases to amaze me, and it's path-mangling tricks are beyond me.=20=

I'm pretty sure that
if we somehow stuff "../Modules/regexmodule.c" (assuming we're building=20=

in a subdir of the main dir)
into the list of module sources we'll get something like=20
build/tmp.blabla/../Modules/regexmodule.o
for the object filename.

Does anyone else feel up to fixing this?
--
- Jack Jansen        <Jack.Jansen@oratrix.com>       =20
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma=20
Goldman -



From Jack.Jansen@cwi.nl  Wed Dec  4 11:15:37 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Wed, 4 Dec 2002 12:15:37 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <11a801c29b80$89bb2e20$0900a8c0@spiff>
Message-ID: <B750EDBC-0779-11D7-93BC-0030655234CE@cwi.nl>

On Wednesday, Dec 4, 2002, at 11:33 Europe/Amsterdam, Fredrik Lundh 
wrote:
> what's wrong with:
>
> #4
>     for path in sys.path:
>         if isinstance(path, types.StringTypes):
>             do_standard_import(...)
>         else:
>             path.do_import(...)
>

After reading Just's later message and the explanation about how Jython 
does it
I think that my preference is now something like the "if hasattr(path, 
'find_module')"
fix. This will allow you to use a subtype of strings on sys.path. Make 
one
pass over sys.path replacing the zipfiles and other funnies by a 
specialised
subtype of string.
		
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From Jack.Jansen@cwi.nl  Wed Dec  4 11:18:58 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Wed, 4 Dec 2002 12:18:58 +0100
Subject: [Python-Dev] Make universal newlines non-optional
In-Reply-To: <002b01c29b81$a4689900$df3fe8d9@mira>
Message-ID: <2F08AD4B-077A-11D7-93BC-0030655234CE@cwi.nl>

On Wednesday, Dec 4, 2002, at 11:41 Europe/Amsterdam, Martin v. L=F6wis=20=

wrote:

>> Well, we don't know yet whether it will work on non-mainstream
>> platforms. It'll probably be okay on BeOS or Amiga or such
>> (and maybe the  maintainers of those platforms track CVS
>> anyway?), but I wouldn't vouch for Windows CE or so.
>
> What would be potential consequences of it not working?

Currently you could simply disable it. If the ifdefs are gone then the=20=

platform
maintainers will have to actually fix it.

But maybe I'm too defensive,
--
- Jack Jansen        <Jack.Jansen@oratrix.com>       =20
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma=20
Goldman -



From mwh@python.net  Wed Dec  4 11:29:10 2002
From: mwh@python.net (Michael Hudson)
Date: 04 Dec 2002 11:29:10 +0000
Subject: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime  picklesize.py,NONE,1.1
In-Reply-To: Tim Peters's message of "Tue, 03 Dec 2002 13:52:55 -0500"
References: <BIEJKCLHCIOIHAGOKOLHAEJBEBAA.tim.one@comcast.net>
Message-ID: <2md6oiqayh.fsf@starship.python.net>

Tim Peters <tim.one@comcast.net> writes:

> [Michael Hudson]
> > Presumably there's a possibility of an optimization for pickling
> > homogeneous (i.e. all the same type) lists (in pickle.py, not here).
> >
> > Hard to say whether it would be worth it, though.
> 
> I don't really care about lists of date objects.  The intent was just to see
> how much of the administrative pickle bloat could be saved by pickle's
> internal memo facility when multiple date objects appear in a structure for
> *whatever* reason (be it a list or tuple of dates, or a dict keyed by dates,
> or an object with multiple data-value attributes, or ...).

OK.

> The administrative pickle bloat for a single date instance is severe:
> 
>     pickling 2000-12-13 via Python -- pickle length 80
>     pickling 2000-12-13 via C -- pickle length 43

Yow.

> The internal pickle memo saves a lot if more than one data instance appears,
> via "remembering" parts of the overhead scaffolding.  But in the end a data
> object has a 4-byte state string, and there's still a lot more overhead than
> state stored in the pickles.

Seems so.

> > ...
> > Here's a fairly simple minded patch to the pickling side of pickle.py:
> > it seems to save about 6 bytes per object in the good cases.
> >
> > with:
> > list of 100 dates via      C -- 1236 bytes, 12.36 bytes/obj
> >
> > without:
> > list of 100 dates via      C -- 1871 bytes, 18.71 bytes/obj
> 
> The "without" number is most curious.  When I run the checked-in code, I get
> 
>   list of 100 dates via      C -- 1533 bytes, 15.33 bytes/obj
> 
> This was w/ CVS Python, Win2K and MSVC6.  Ah!  You must have fiddled the
> script to use pickle instead of cPickle.

Yep.

> I get 1871 bytes then.  I'm surprised they're so different.

I know cPickle does some tricks if refcounts are 1 that aren't
duplicated in pickle.py.

> > I'm not going to pursue this further unless someone thinks it's a
> > worthwhile move.
> 
> I don't personally have large pickles of homogeneous lists, so hard to say.

Me neither.

> The C implementation of pickle would also need to be fiddled.

That was part of "pursuing this further" (as is, e.g., unpickling
support).

> Cutting the pickle overheads for instances of new-style classes
> would more clearly be worthwhile.

Hard to see how to do that.  You could reduce the overhead by giving
_datetime_unpickler a shorter name :-)

In general, though, any improvements we can add here are likely to be
blown away by ramming the pickle through zlib, so I'm not sure this is
worth worrying about too much.

Cheers,
M.

-- 
  And then the character-only displays went away (leading to
  increasingly silly graphical effects and finally to ads on
  web pages).                      -- John W. Baxter, comp.lang.python


From just@letterror.com  Wed Dec  4 11:52:39 2002
From: just@letterror.com (Just van Rossum)
Date: Wed,  4 Dec 2002 12:52:39 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <11a801c29b80$89bb2e20$0900a8c0@spiff>
Message-ID: <r01050400-1022-F0395136077E11D7ADDD003065D5E7E4@[10.0.0.23]>

Fredrik Lundh wrote:

> what's wrong with:
> 
> #4
>     for path in sys.path:
>         if isinstance(path, types.StringTypes):
>             do_standard_import(...)
>         else:
>             path.do_import(...)

That, or what Jack said, which I think is even better:

#5
    for path in sys.path:
        if hasattr(path, "find_module"):
            loader = path.find_module(name)
        if isinstance(path, types.StringTypes):
            do_standard_import(...)
        else:
            pass

(I originally thought it might be important that path elements be str
subclasses, but I'm not so sure anymore. Especially after I wrote a str subclass
in C: it's a HUGE pain ;-)

Just


From mwh@python.net  Wed Dec  4 13:00:40 2002
From: mwh@python.net (Michael Hudson)
Date: 04 Dec 2002 13:00:40 +0000
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: Jack Jansen's message of "Wed, 4 Dec 2002 12:02:07 +0100"
References: <D464FBC2-0777-11D7-93BC-0030655234CE@cwi.nl>
Message-ID: <2m7keqq6pz.fsf@starship.python.net>

Jack Jansen <Jack.Jansen@cwi.nl> writes:

> I am tempted to think that this problem has always existed, but has=20
> gone unnoticed until now.

It's definitely gotten worse recently.

> Now that I know about the problem it clears up some strange behaviour=20
> I've seen in the past.
> I would love to see this fixed, but I'm afraid I'm not the person to do=20=
> 
> it. Distutils
> never ceases to amaze me, and it's path-mangling tricks are beyond me.=20=
> 
> I'm pretty sure that
> if we somehow stuff "../Modules/regexmodule.c" (assuming we're building=20=
> 
> in a subdir of the main dir)
> into the list of module sources we'll get something like=20
> build/tmp.blabla/../Modules/regexmodule.o
> for the object filename.
> 
> Does anyone else feel up to fixing this?

I'll have a look, but so far I'm only confusing myself.

The first out-of-tree build I did today put one .o file in the wrong
place (bsddbmodule.o, I think).  Now all of them are ending up in the
wrong place.

Cheers,
M.

-- 
 Very clever implementation techniques are required to implement this
 insanity correctly and usefully, not to mention that code written
 with this feature used and abused east and west is exceptionally
 exciting to debug.       -- Erik Naggum on Algol-style "call-by-name"


From mwh@python.net  Wed Dec  4 13:26:55 2002
From: mwh@python.net (Michael Hudson)
Date: 04 Dec 2002 13:26:55 +0000
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: Michael Hudson's message of "04 Dec 2002 13:00:40 +0000"
References: <D464FBC2-0777-11D7-93BC-0030655234CE@cwi.nl> <2m7keqq6pz.fsf@starship.python.net>
Message-ID: <2m4r9trk2o.fsf@starship.python.net>

Michael Hudson <mwh@python.net> writes:

> The first out-of-tree build I did today put one .o file in the wrong
> place (bsddbmodule.o, I think).  Now all of them are ending up in the
> wrong place.

Not sure how I managed that one, but I've found the offending checkin:

----------------------------
revision 1.52
date: 2002/11/14 01:29:00;  author: akuchling;  state: Exp;  lines: +3 -3
[Bug #599248] ext module generation problem

If you have source files srcdir1/foo.c and srcdir2/foo.c, the
temporary .o for both files is written to build/temp.<platform>/foo.o.
This patch sets strip_dir to false for both calls to object_filename,
so now the object files are written to temp.<platform>/srcdir1/foo.o
and .../srcdir2/foo.o.

2.2 bugfix candidate
----------------------------

Now to see if patching it up is easy...

Cheers,
M.

-- 
  ARTHUR:  Ford, you're turning into a penguin, stop it.
                    -- The Hitch-Hikers Guide to the Galaxy, Episode 2


From Paul.Moore@atosorigin.com  Wed Dec  4 13:27:38 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 4 Dec 2002 13:27:38 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E4E@UKDCX001.uk.int.atosorigin.com>

From: Martin v. L=F6wis [mailto:martin@v.loewis.de]
> Just van Rossum <just@letterror.com> writes:

>> I'm working hard on a zipimport module that uses
>> sys.import_hooks. It's going to be SO much cleaner than
>> the existing zip import patch.

> Ok. I then won't apply the existing patch until
> a) your zipfile patch comes along to review, or
> b) significant time has passed after 2.3a1 (for some
>    reasonable value for "significant")

The only reservation I have about this is if the existing patch
starts going "out of date" again. I can't get CVS to work on my
slow dial-up connection, for anything other than a straight cvs
checkout, so I'm reduced to full downloads and manually updating
the patch. This is both tedious and error-prone.

Just promised his patch in "a couple of days". If that's the
timescale, I'm fine with this plan. But if it's a couple of weeks,
then I'm going to be annoyed if the reintegration turns out to be
complicated.

Paul.


From Paul.Moore@atosorigin.com  Wed Dec  4 14:06:28 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 4 Dec 2002 14:06:28 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861996C@UKDCX001.uk.int.atosorigin.com>

> My current sys.import_hooks patch implements #3 and I think
> that's the most logical thing to do; after all sys.path may not
> be meaningful for a hook. #2 makes little sense I guess. #1
> could be an option, but, given the current state of import.c,
> would be a lot harder to implement. I also think that #3 is the
> most efficient scheme, as the hook can save some local state
> while traversing sys.path.

Have you looked at Gordon McMillan's import utilities? He handles
this by having a "shadowpath" in the manager which maps sys.path
entries to "Owners", where each Owner is responsible for loading
from one location (directory, zipfile, whatever).

I'm convinced you're reinventing the wheel here, at least in part.
(But reinventing it in C, rather than Python, so it's not a
completely bad thing :-))

Paul.


From just@letterror.com  Wed Dec  4 14:51:38 2002
From: just@letterror.com (Just van Rossum)
Date: Wed,  4 Dec 2002 15:51:38 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996C@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-EA3BD678079711D7ADDD003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> Have you looked at Gordon McMillan's import utilities? He handles
> this by having a "shadowpath" in the manager which maps sys.path
> entries to "Owners", where each Owner is responsible for loading
> from one location (directory, zipfile, whatever).
> 
> I'm convinced you're reinventing the wheel here, at least in part.
> (But reinventing it in C, rather than Python, so it's not a
> completely bad thing :-))

In the meantime things have changed: death to sys.import_hooks, and yay to
arbitrary objects on sys.path. I think this is more or less the same as iu.py,
except that there's no need for a "shadowpath" (if not a string, the sys.path
item _is_ the "owner").

Just


From Paul.Moore@atosorigin.com  Wed Dec  4 14:55:16 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 4 Dec 2002 14:55:16 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E51@UKDCX001.uk.int.atosorigin.com>

From: Just van Rossum [mailto:just@letterror.com]
> In the meantime things have changed: death to sys.import_hooks,
> and yay to arbitrary objects on sys.path. I think this is more
> or less the same as iu.py, except that there's no need for a
> "shadowpath" (if not a string, the sys.path item _is_ the
> "owner").

How do I put a zipfile on my PYTHONPATH environment variable?

Paul.


From Paul.Moore@atosorigin.com  Wed Dec  4 15:06:51 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 4 Dec 2002 15:06:51 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E52@UKDCX001.uk.int.atosorigin.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C29BA6.C61AC446
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

From: Just van Rossum [mailto:just@letterror.com]
> I think this is more or less the same as iu.py,
> except that there's no need for a "shadowpath"

Thinking further, what is the advantage of your code
over iu.py, if this is true? Performance? Do you have
figures?

Paul.

PS I attach a small test script I used to get a feel
   for the performance of Jim's patch. Unzip this file,
   then do
       cd test
       zip -0r ../test0 *
       zip -9r ../test9 *
   Create a copy of the standard library as Lib, and do
       cd Lib
       zip -0r ../stdlib0 *
       zip -9r ../stdlib9 *
   Then run test.py.

   Figures for me, with CVS + Jim's patch, on a 450MHz
   Athlon with 256M RAM running Windows 2000, were:

       Lib [[['Lib\\email']]]:    0.309
       stdlib0.zip [[['stdlib0.zip\\email']]]:    0.866
       stdlib9.zip [[['stdlib9.zip\\email']]]:    1.154
       test [[test\testCR.pyc]]:    0.344
       test0.zip [[test0.zip\testCR.py]]:    0.204
       test9.zip [[test9.zip\testCR.py]]:    0.348

   So import from zip is noticeably slower than directory
   imports on big zips, but *faster* on small ones.
   Compressed zipfiles are slower than uncompressed.

------_=_NextPart_001_01C29BA6.C61AC446
Content-Type: application/x-zip-compressed;
	name="test.zip"
Content-Transfer-Encoding: base64
Content-Description: test.zip
Content-Disposition: attachment;
	filename="test.zip"

UEsDBAoAAAAAAF2lgy0AAAAAAAAAAAAAAAAFABEAdGVzdC9VVA0AB1IX7T0A9Os9URftPVBLAwQU
AAIACAA4l4MtKBXJLB8AAAAiAAAADgARAHRlc3QvdGVzdENSLnB5VVQNAAes/uw9APTrPVEX7T1L
SU1TKEktLtHQtOJVAIKCosy8EgUlzzywqHOQEi8AUEsDBBQAAgAIACaXgy2Voha5IwAAACYAAAAQ
ABEAdGVzdC90ZXN0Q1JMRi5weVVUDQAHiP7sPQD06z1SF+09S0lNUyhJLS7R0LTi5VIAgoKizLwS
BSXPPLCwc5CPmxIvFwBQSwMEFAACAAgAMZeDLd8DWIcfAAAAIgAAAA4AEQB0ZXN0L3Rlc3RMRi5w
eVVUDQAHnv7sPQD06z1TF+09S0lNUyhJLS7R0LTiUgCCgqLMvBIFJc88sKiPmxIXAFBLAwQUAAIA
CABGl4MtIqMenSAAAAAjAAAADwARAHRlc3QvdGVzdFBZQy5weVVUDQAHxP7sPQD06z1UF+09S0lN
UyhJLS7R0LTiUgCCgqLMvBIFJc88sGhApLMSFwBQSwMEFAACAAgAhKSDLcEhu3RtAAAA6gAAABAA
EQB0ZXN0L3Rlc3RQWUMucHljVVQNAAe4Fe09APTrPVUX7T2z/sTLdejfG9tkBihghNLFvEAihYGh
hYEhCshgZAjWYAKKoKhjBqnjZABLu3sAFUPU+BVzAUnPPIWS1OKSgEhnDZB6NKKYH0iA5GOgivQK
KotZoGIgs4uZwLb4aYA5MBk0Hn7jQGrtwRpAjgYAUEsDBBQAAgAIAAelgy02cU8gcAEAAEcEAAAH
ABEAdGVzdC5weVVUDQAHrhbtPQD06z1pF+09tVLLasMwELwb/A9bg7ENqUnppQ3kFMgph9JbMUYo
sdyIyJaRVEry9dVafgUnh0Ir8EOzu7MzK/GqkcqAPmvfI/ZNGmqOsEYgxd9slfue7xWsBMU0M5Us
vgTTcbLyPbCrlApqWjHgdVvTxdMTO49JuHjp8h7WENnECGhdjMhF8H0HSdPCqTZUGf3NzTGOCImm
XLgKJqYNM6zJXUqv3doYLHU+SiUrMBz1OuMHIQ+n3qFh2lghscX6dtemO35UZsnb2gEcm2YW76R0
XVhFubhCLIcZJ8hxfIrWnyx+Wi6nVhUTkhZxS5DMYGTp+7PDKAkencj7HhrFawNBqCHLQp3nKwhf
0ucygBDQ/sJJTkk7PEIWyJ9M54TP3w8KWTfvM2i3vZF1A3z72PxiqI4muYnvtsmd/HsR2/ufjsL1
tWdRcsGmZ4EubQb6jIMd3wcLCLQp7BVephfejNvXdtuPYHrNZzQYxEL8Diy4mXMMV8D3fgBQSwEC
FgsKAAAAAABdpYMtAAAAAAAAAAAAAAAABQAJAAAAAAAAABAA/0EAAAAAdGVzdC9VVAUAB1IX7T1Q
SwECFgsUAAIACAA4l4MtKBXJLB8AAAAiAAAADgAJAAAAAAABACAAtoE0AAAAdGVzdC90ZXN0Q1Iu
cHlVVAUAB6z+7D1QSwECFgsUAAIACAAml4MtlaIWuSMAAAAmAAAAEAAJAAAAAAABACAAtoGQAAAA
dGVzdC90ZXN0Q1JMRi5weVVUBQAHiP7sPVBLAQIWCxQAAgAIADGXgy3fA1iHHwAAACIAAAAOAAkA
AAAAAAEAIAC2gfIAAAB0ZXN0L3Rlc3RMRi5weVVUBQAHnv7sPVBLAQIWCxQAAgAIAEaXgy0iox6d
IAAAACMAAAAPAAkAAAAAAAEAIAC2gU4BAAB0ZXN0L3Rlc3RQWUMucHlVVAUAB8T+7D1QSwECFgsU
AAIACACEpIMtwSG7dG0AAADqAAAAEAAJAAAAAAAAACAAtoGsAQAAdGVzdC90ZXN0UFlDLnB5Y1VU
BQAHuBXtPVBLAQIWCxQAAgAIAAelgy02cU8gcAEAAEcEAAAHAAkAAAAAAAEAIAC2gVgCAAB0ZXN0
LnB5VVQFAAeuFu09UEsFBgAAAAAHAAcA2AEAAP4DAAAAAA==

------_=_NextPart_001_01C29BA6.C61AC446--


From skip@pobox.com  Wed Dec  4 15:17:18 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 4 Dec 2002 09:17:18 -0600
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: <D464FBC2-0777-11D7-93BC-0030655234CE@cwi.nl>
References: <m365uby0r3.fsf@mira.informatik.hu-berlin.de>
 <D464FBC2-0777-11D7-93BC-0030655234CE@cwi.nl>
Message-ID: <15854.7294.853169.73266@montanaro.dyndns.org>

    Jack> Distutils never ceases to amaze me, and it's path-mangling tricks
    Jack> are beyond me.

:-)

    Jack> I'm pretty sure that if we somehow stuff
    Jack> "../Modules/regexmodule.c" (assuming we're building in a subdir of
    Jack> the main dir) into the list of module sources we'll get something
    Jack> like build/tmp.blabla/../Modules/regexmodule.o for the object
    Jack> filename.

    Jack> Does anyone else feel up to fixing this?

I poked around a bit yesterday but didn't get too far.  Deep distutils
pondering is beyond my feeble brain.  In PyBuildExt.build_extensions() in
setup.py, you will see os.getcwd() tacked onto the front of the generated
path in a couple places.  That appears to be the cause of the absolute paths
we see.  I think removing it will be a necessary, but not sufficient,
condition for a fix.  Removing it will create a relative path, but I think
it will still be incorrect.  One other observation: in a separate build
directory no Mac/Modules directory is created (though Mac/Python is).  This
is presumably where the Mac .o files belong.

Skip



From just@letterror.com  Wed Dec  4 15:12:37 2002
From: just@letterror.com (Just van Rossum)
Date: Wed,  4 Dec 2002 16:12:37 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E51@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-B7774702079B11D7ADDD003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> From: Just van Rossum [mailto:just@letterror.com]
> > In the meantime things have changed: death to sys.import_hooks,
> > and yay to arbitrary objects on sys.path. I think this is more
> > or less the same as iu.py, except that there's no need for a
> > "shadowpath" (if not a string, the sys.path item _is_ the
> > "owner").
> 
> How do I put a zipfile on my PYTHONPATH environment variable?

We do what Jython does: somewhere during initialisation, *.zip sys.path items
will be replaced by zipimporter instances.

Just


From just@letterror.com  Wed Dec  4 15:18:46 2002
From: just@letterror.com (Just van Rossum)
Date: Wed,  4 Dec 2002 16:18:46 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E52@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-B26D59B8079B11D7ADDD003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> From: Just van Rossum [mailto:just@letterror.com]
> > I think this is more or less the same as iu.py,
> > except that there's no need for a "shadowpath"
> 
> Thinking further, what is the advantage of your code
> over iu.py, if this is true?

Simplicity. I suspect that the shadowpath is only needed if sys.path items must
be strings. I could be wrong, though.

And it's more or less what Jython does, which is nice. It's simple and works
well. (Samuele: pls correct me if I'm wrong ;-)

> Performance? Do you have figures?

No and no ;-)

(Later, later: the new hook already works, now I need to get a proof-of-concept
of zipimport going.)

Just


From Paul.Moore@atosorigin.com  Wed Dec  4 16:28:02 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 4 Dec 2002 16:28:02 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E57@UKDCX001.uk.int.atosorigin.com>

From: Just van Rossum [mailto:just@letterror.com]
> We do what Jython does: somewhere during initialisation,
> *.zip sys.path items will be replaced by zipimporter
> instances.

BTW, on Gordon's site, in the iu documentation, he mentions
that "the assumption that sys.path is occupied solely by
strings seems ineradicable". I don't know if he's got any
specific issues in mind, but it might be worth checking out
(get the screams of protest out of the way before the code
goes in :-))

I've copied Gordon in case he can give any specifics...

Paul.


From gmcm@hypernet.com  Wed Dec  4 17:07:32 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Wed, 4 Dec 2002 12:07:32 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB885E2E57@UKDCX001.uk.int.atosorigin.com>
Message-ID: <3DEDF004.25379.630FD638@localhost>

On 4 Dec 2002 at 16:28, Moore, Paul wrote:

> BTW, on Gordon's site, in the iu documentation, he
> mentions that "the assumption that sys.path is
> occupied solely by strings seems ineradicable". I
> don't know if he's got any specific issues in mind,
> but it might be worth checking out (get the screams
> of protest out of the way before the code goes in
> :-)) 
> 
> I've copied Gordon in case he can give any
> specifics... 

Code like this:
 for p in sys.path:
   x = os.path.join(p, ...)
   ....
is very common (I patched linecache.py for this
after imputil went into the std lib). Since PYTHONPATH
can consist only of strings, it seems wise to 
tackle the issue (dealing with strings that describe
non-directory collections of modules) instead of postponing it. Also seems sensible to make it
so that if X works on PYTHONPATH, 
sys.path.append(X) should work, too.

The shadowpath consists only of objects, and
len(shadowpath) <= len(sys.path); it's either
lazy or caching, depending on POV.

[There's a metapath, too. That lets you put
overrides in front. You can even override
frozen and builtin modules.]

-- Gordon
http://www.mcmillan-inc.com/



From martin@v.loewis.de  Wed Dec  4 19:56:15 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 04 Dec 2002 20:56:15 +0100
Subject: [Python-Dev] Make universal newlines non-optional
In-Reply-To: <2F08AD4B-077A-11D7-93BC-0030655234CE@cwi.nl>
References: <2F08AD4B-077A-11D7-93BC-0030655234CE@cwi.nl>
Message-ID: <m3n0nlpnhc.fsf@mira.informatik.hu-berlin.de>

Jack Jansen <Jack.Jansen@cwi.nl> writes:

> Currently you could simply disable it. If the ifdefs are gone then the
> platform
> maintainers will have to actually fix it.
> 
> But maybe I'm too defensive,

Ok. Then I'd propose to leave this for 2.3, and remove the option for
2.4.

Regards,
Martin


From martin@v.loewis.de  Wed Dec  4 19:58:58 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 04 Dec 2002 20:58:58 +0100
Subject: [Python-Dev] extension module .o files wind up in the wrong place
In-Reply-To: <15854.7294.853169.73266@montanaro.dyndns.org>
References: <m365uby0r3.fsf@mira.informatik.hu-berlin.de>
 <D464FBC2-0777-11D7-93BC-0030655234CE@cwi.nl>
 <15854.7294.853169.73266@montanaro.dyndns.org>
Message-ID: <m3isy9pnct.fsf@mira.informatik.hu-berlin.de>

Skip Montanaro <skip@pobox.com> writes:

> I poked around a bit yesterday but didn't get too far.  Deep distutils
> pondering is beyond my feeble brain.  In PyBuildExt.build_extensions() in
> setup.py, you will see os.getcwd() tacked onto the front of the generated
> path in a couple places.  That appears to be the cause of the absolute paths
> we see.  I think removing it will be a necessary, but not sufficient,
> condition for a fix.  

I now believe that Michael's analysis is correct: I.e. it is fine that
setup.py produces absolute paths, since distutils is supposed to make
them relative again.

Regards,
Martin



From just@letterror.com  Wed Dec  4 21:12:48 2002
From: just@letterror.com (Just van Rossum)
Date: Wed,  4 Dec 2002 22:12:48 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEDF004.25379.630FD638@localhost>
Message-ID: <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]>

Gordon McMillan wrote:

> Code like this:
>  for p in sys.path:
>    x = os.path.join(p, ...)
>    ....
> is very common (I patched linecache.py for this after imputil went
> into the std lib). Since PYTHONPATH can consist only of strings, it
> seems wise to tackle the issue (dealing with strings that describe
> non-directory collections of modules) instead of postponing it. Also
> seems sensible to make it so that if X works on PYTHONPATH,
> sys.path.append(X) should work, too.

Erm, I'm not sure what you're saying... Are you saying that we should fix all
cases where non-strings on sys.path cause problems, or are you saying that
there's so much code out there assuming sys.path contains strings, and that we
therefore should stick with strings?

Both positions can be defended, and both have their problems.

A) Stick with strings. Hooks can be implemented by subclassing str. This is
great for hooks written in Python, but subclassing str in C is not
straightforward. Things can still break, though: eg.
os.path.basename(strsubinst) will return a regular string, not an instance of
the subclass; might be an issue.

B) Allow arbitrary objects on sys.path. Hooks are then easier to write (in C),
but some code breakage will occur. The std library we can fix (if needed), but
third-party code might break.

I would very much prefer B, but if it turns out that we can't break the string
assumption, I'd still be happy with A (rather that than nothing!).

Regarding PYTHONPATH and sys.path.append("/path/to/my/archive.zip"): for now I'd
suggest that the sys.path traversing code checks for a .zip extension, and
replace the item with a zipimporter instance. This check can be very cheap.
Later we could add a general extension-checking feature, where one could
register an import hook for a specific extension. This might be a case of YAGNI,
though...

Just


From skip@pobox.com  Wed Dec  4 21:22:48 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 4 Dec 2002 15:22:48 -0600
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]>
References: <3DEDF004.25379.630FD638@localhost>
 <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]>
Message-ID: <15854.29224.574966.648231@montanaro.dyndns.org>

    Just> Are you saying that we should fix all cases where non-strings on
    Just> sys.path cause problems, or are you saying that there's so much
    Just> code out there assuming sys.path contains strings, and that we
    Just> therefore should stick with strings?

    Just> Both positions can be defended, and both have their problems.

    ...

I think there is a third position: make sys.path magic somehow (but have it
still (appear to) be a list of strings) so that changes to it affect a
behind-the-scenes list of objects which is normally used to do path-ish
stuff.

Skip


From mal@lemburg.com  Wed Dec  4 21:56:12 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 04 Dec 2002 22:56:12 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <3DEDF004.25379.630FD638@localhost>        <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org>
Message-ID: <3DEE79FC.1020802@lemburg.com>

Skip Montanaro wrote:
>     Just> Are you saying that we should fix all cases where non-strings on
>     Just> sys.path cause problems, or are you saying that there's so much
>     Just> code out there assuming sys.path contains strings, and that we
>     Just> therefore should stick with strings?
> 
>     Just> Both positions can be defended, and both have their problems.
> 
>     ...
> 
> I think there is a third position: make sys.path magic somehow (but have it
> still (appear to) be a list of strings) so that changes to it affect a
> behind-the-scenes list of objects which is normally used to do path-ish
> stuff.

Instead of making imports even more magical, why not do what
the subject line says: add more useful hooks to the existing
C implementation of the import statement, e.g. provide hook
callbacks to manage how the import mechanism looks up, finds
and opens files and callbacks to manage extracting module
data from the found entities.

The only drawback I see for the current import hook
situation is that each hook has to implement *all* the
import functionality, since there's only one entry point:
the top-level import statement. The existing machinery could
be put to much better use (w/r to adding support for e.g.
ZIP files) by adding more callbacks in appropriate places.

Anyway, have fun,
-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From DavidA@ActiveState.com  Wed Dec  4 22:02:47 2002
From: DavidA@ActiveState.com (David Ascher)
Date: Wed, 04 Dec 2002 14:02:47 -0800
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEDF004.25379.630FD638@localhost>
References: <3DEDF004.25379.630FD638@localhost>        <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org>
Message-ID: <3DEE7B87.5090602@ActiveState.com>

Skip Montanaro wrote:

>     Just> Are you saying that we should fix all cases where non-strings on
>     Just> sys.path cause problems, or are you saying that there's so much
>     Just> code out there assuming sys.path contains strings, and that we
>     Just> therefore should stick with strings?
>
>     Just> Both positions can be defended, and both have their problems.
>
>     ...
>
> I think there is a third position: make sys.path magic somehow (but have it
> still (appear to) be a list of strings) so that changes to it affect a
> behind-the-scenes list of objects which is normally used to do path-ish
> stuff.

I'm not sure I understand Skip's proposal, which may be the same as the 
following, which strives for backwards compatibility:

   - Define an alternative path which can have non-strings on it, and define
     sys.path to be a "view" of the string elements in this superpath.

IOW:

    assert sys.path == ['a', 'b', 'c']
    assert sys.superpath == ['a', 'b', 'c']
    sys.superpath.insert(0, CodeGenerator())
    assert sys.superpath == [<CodeGenerator instance>, 'a', 'b', 'c']
    assert sys.path == ['a', 'b', 'c']
    sys.path.insert(0, 'foo')
    assert sys.superpath == ['foo', <CodeGenerator instance>, 'a', 'b', 'c']
    assert sys.path == ['foo', 'a', 'b', 'c']

the superpath is used in by the import mechanism, and modifications to 
sys.path propagate back.

It's not ideal, but I think it's backwards compatible.

--da




From skip@pobox.com  Wed Dec  4 22:12:00 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 4 Dec 2002 16:12:00 -0600
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEE7B87.5090602@ActiveState.com>
References: <3DEDF004.25379.630FD638@localhost>
 <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]>
 <15854.29224.574966.648231@montanaro.dyndns.org>
 <3DEE7B87.5090602@ActiveState.com>
Message-ID: <15854.32176.422808.960868@montanaro.dyndns.org>

    >> I think there is a third position: make sys.path magic somehow (but
    >> have it still (appear to) be a list of strings) so that changes to it
    >> affect a behind-the-scenes list of objects which is normally used to
    >> do path-ish stuff.

    David> I'm not sure I understand Skip's proposal, which may be the same
    David> as the following, which strives for backwards compatibility:

    David>    - Define an alternative path which can have non-strings on it,
    David>      and define sys.path to be a "view" of the string elements in
    David>      this superpath.

Yeah, somethin' like that...

Skip


From rjones@ekit-inc.com  Wed Dec  4 22:06:11 2002
From: rjones@ekit-inc.com (Richard Jones)
Date: Thu, 5 Dec 2002 09:06:11 +1100
Subject: [Python-Dev] PEP 301: Package Index and Metadata for Distutils
In-Reply-To: <3DEDD21C.9030909@lemburg.com>
References: <200212041547.14551.rjones@ekit-inc.com> <3DEDD21C.9030909@lemburg.com>
Message-ID: <200212050906.11044.rjones@ekit-inc.com>

On Wed, 4 Dec 2002 8:59 pm, M.-A. Lemburg wrote:
> One nit: in order to make the setup.py script work for Python 1.5.2
> and up, you should provide a way which let's the distutils
> versions of those platforms ignore the classification keyword
> argument for setup().

The documentation of the classifiers keyword will include a specific warning 
about backwards compatibility. If there no objections, I'll include a code 
fragment that setup.py authors may use so their setup.py will run under 
python <2.2.3.


    Richard



From just@letterror.com  Wed Dec  4 22:32:26 2002
From: just@letterror.com (Just van Rossum)
Date: Wed,  4 Dec 2002 23:32:26 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEE7B87.5090602@ActiveState.com>
Message-ID: <r01050400-1022-4654686407D811D7AD06003065D5E7E4@[10.0.0.23]>

David Ascher wrote:

> I'm not sure I understand Skip's proposal, which may be the same as the 
> following, which strives for backwards compatibility:
> 
>    - Define an alternative path which can have non-strings on it, and define
>      sys.path to be a "view" of the string elements in this superpath.
> 
> IOW:
> 
>     assert sys.path == ['a', 'b', 'c']
>     assert sys.superpath == ['a', 'b', 'c']
>     sys.superpath.insert(0, CodeGenerator())
>     assert sys.superpath == [<CodeGenerator instance>, 'a', 'b', 'c']
>     assert sys.path == ['a', 'b', 'c']
>     sys.path.insert(0, 'foo')
>     assert sys.superpath == ['foo', <CodeGenerator instance>, 'a', 'b', 'c']
>     assert sys.path == ['foo', 'a', 'b', 'c']
> 
> the superpath is used in by the import mechanism, and modifications to 
> sys.path propagate back.
> 
> It's not ideal, but I think it's backwards compatible.

How is this different (besides being more complex <wink>) from making import
hooks string subclasses?

(I'd still like to see more examples of code breaking if a sys.path item isn't a
string. Not that I don't believe it, but I'd like to get an impression of the
severeness of the damage.)

Just


From pedronis@bluewin.ch  Wed Dec  4 22:23:09 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Wed, 4 Dec 2002 23:23:09 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <3DEDF004.25379.630FD638@localhost>        <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com>
Message-ID: <037001c29be3$ba396400$6d94fea9@newmexico>

From: "David Ascher" <DavidA@ActiveState.com>
> Skip Montanaro wrote:
>
> >     Just> Are you saying that we should fix all cases where non-strings on
> >     Just> sys.path cause problems, or are you saying that there's so much
> >     Just> code out there assuming sys.path contains strings, and that we
> >     Just> therefore should stick with strings?
> >
> >     Just> Both positions can be defended, and both have their problems.
> >
> >     ...
> >
> > I think there is a third position: make sys.path magic somehow (but have it
> > still (appear to) be a list of strings) so that changes to it affect a
> > behind-the-scenes list of objects which is normally used to do path-ish
> > stuff.
>
> I'm not sure I understand Skip's proposal, which may be the same as the
> following, which strives for backwards compatibility:
>
>    - Define an alternative path which can have non-strings on it, and define
>      sys.path to be a "view" of the string elements in this superpath.
>
> IOW:
>
>     assert sys.path == ['a', 'b', 'c']
>     assert sys.superpath == ['a', 'b', 'c']
>     sys.superpath.insert(0, CodeGenerator())
>     assert sys.superpath == [<CodeGenerator instance>, 'a', 'b', 'c']
>     assert sys.path == ['a', 'b', 'c']
>     sys.path.insert(0, 'foo')
>     assert sys.superpath == ['foo', <CodeGenerator instance>, 'a', 'b', 'c']
>     assert sys.path == ['foo', 'a', 'b', 'c']
>
> the superpath is used in by the import mechanism, and modifications to
> sys.path propagate back.
>
> It's not ideal, but I think it's backwards compatible.
>

I would it support this kind of code:

sys.path = L
L.append(...) ?

regards.



From fredrik@pythonware.com  Wed Dec  4 22:42:14 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 4 Dec 2002 23:42:14 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-4654686407D811D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <028701c29be6$665c3ee0$ced241d5@hagrid>

Just van Rossum wrote:

> (I'd still like to see more examples of code breaking if a sys.path item isn't a
> string. Not that I don't believe it, but I'd like to get an impression of the
> severeness of the damage.)

>From the ImageFont module in PIL 1.1.3:

def load_path(filename):
    "Load a font file, searching along the Python path."
    for dir in sys.path:
        try:
            return load(os.path.join(dir, filename))
        except IOError:
            pass
    raise IOError, "cannot find font file"

(this is fixed in 1.1.4)

FWIW, I've done lots of stuff that scans sys.path directories for
various purposes.  It would probably be a good idea to use a string
subclass for the zip importer, but I don't think we need to make
stringness an absolute requirement.

Let's make it *really* easy to use zip archives, and reasonably easy
to use other custom importers.

</F>



From just@letterror.com  Wed Dec  4 22:49:30 2002
From: just@letterror.com (Just van Rossum)
Date: Wed,  4 Dec 2002 23:49:30 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <028701c29be6$665c3ee0$ced241d5@hagrid>
Message-ID: <r01050400-1022-A8FC5EEC07DA11D7AD06003065D5E7E4@[10.0.0.23]>

Fredrik Lundh wrote:

> FWIW, I've done lots of stuff that scans sys.path directories for
> various purposes.  It would probably be a good idea to use a string
> subclass for the zip importer, but I don't think we need to make
> stringness an absolute requirement.

+1

> Let's make it *really* easy to use zip archives, and reasonably easy
> to use other custom importers.

+1

Just


From bbum@codefab.com  Wed Dec  4 23:25:53 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Wed, 4 Dec 2002 18:25:53 -0500
Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies
Message-ID: <BBA757DD-07DF-11D7-B2E7-000393877AE4@codefab.com>

I'm writing a Transport class to handle proxy servers such that the 
xmlrpclib module will be able to communicate with a server that is on 
the other side of a proxy somewhere.

Most of the difficulty in doing so is the need to navigate the urllib 
vs. urllib2 vs. httplib maze.   I believe I can do what I need entirely 
via urllib, but may use urllib2 because the capabilities better fit the 
needs of my ultimate use.

However, I would like to create a Transport class/subclass that can be 
dropped into the existing xmlrpclib module in the core such that this 
problem is solved once and for all.

As such, I would really like some feedback from the python-dev 
community as to ensure that the efforts I put forth to solve  my 
particular needs are applicable outside of those needs...

thanks!

b.bum



From DavidA@ActiveState.com  Wed Dec  4 23:49:34 2002
From: DavidA@ActiveState.com (David Ascher)
Date: Wed, 04 Dec 2002 15:49:34 -0800
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEDF004.25379.630FD638@localhost>
References: <3DEDF004.25379.630FD638@localhost>        <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico>
Message-ID: <3DEE948E.3010802@ActiveState.com>

Samuele Pedroni wrote:

> I would it support this kind of code:
>
> sys.path = L
> L.append(...) ?

Sure -- sys.path wouldn't be a "real" list object, just one that behaved as one.

Just:

 > How is this different (besides being more complex <wink>) from making
 > import hooks string subclasses?

It doesn't require the import hook to be a string subclass, which you said was 
hard to do if the import hook is to be written in C.  If making it easy to 
write a C import hook isn't a requirement anymore, let me know!

--david



From just@letterror.com  Thu Dec  5 01:00:23 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 02:00:23 +0100
Subject: [Python-Dev] New Import Hook & New Zip Importer
Message-ID: <r01050400-1022-F8B266A507EC11D7AD06003065D5E7E4@[10.0.0.23]>

--643623-2675965952-3248042441=:9000
Content-Type: text/plain; Charset=US-ASCII
Content-Transfer-Encoding: 7bit

Phew, it works.

The attached tgz file contains:
    zipper.patch: patches setup.py, Python/import.c, Python/importdl.h
    zipimport.c: zipimport module, move to Modules/

Do I need to mention it's a rough patch <wink>?

import.c changes:
    - Allow arbitrary objects on sys.path.
    - If a path entry ends in ".zip", replace that entry with
      a zipimporter instance. (obviously, the zipimport module
      must be available for this to work.)
    - The search algo is:
        [handle builtin modules and frozen]
        for p on sys.path:
            if hasattr(p, "find_module"):
                handler = p.find_module(name)
                if handler is None:
                    continue
            if isinstance(p, str):
                [do builtin import]

zipimport.c implements:
    - The zipimporter object; explicit use example:
          sys.path.append(zipimport.zipimporter("myarchive.zip"))
      this is effectively equivalent to
          sys.path.append("myarchive.zip")
    - the zipimporter has two methods:
        find_module(subname) -> None or self
        load_module(fullname) -> module

import.c todo/notes:
    - clean up, refactor, debug, document
    The patch adds two new static functions, find_module2() and
    load_module2(). Ideally these should _replace_ find_module and
    load_module, but since the signature of the new ones is different
    I decided to keep the originals (which are reimplemented to call
    the new ones) to keep the patch small for easier review.

zipimport.c todo:
    - rework the zipimporter class as a subclass of str
    - doco
    - add a get_source(fullname) method (linecache.py could use that!)

Have fun!

Just
--643623-2675965952-3248042441=:9000
Content-Type: application/octet-stream; Name="zipper.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; Filename="zipper.tgz"

H4sIAAAAAAAAA+w8a3fbRq79Kv2KiXJqSzbtSPIjibXJHsWWG986lo/tdNNmc3ho
amRxI5EMScVxG//3C2CefEhWbl8f7uq0kTkcYDAABgNgMPo1iGOePPnuz/y027vt
p3t78N1uP90X3+3urvgWn++gw9Odvc7+/u7T79qdTnf/6Xds70+lSn7maeYljH33
H/hjWT/oNh7/FQT9tZ9fhfzhK5jFUZJt+3/8GO1Ou72/u7tA/rud9tPdgvx3dp/u
fMfafzwp5c//c/k/DkJ/Oh9x1ji/yyZRuD1p1E1bmiVzP5vx2TVP8m+idMTHab5t
5iXpxJvmG/1oFgdTjo31+mMACkLOzn92B++uWNtqOKSWjtUypJaubnnVP6KWrQ5g
AnFkgc+CMGMp9xJ/4kbJiCfvP7AX7DeJzVFIHDngfc+Gm3ifufvrNLgGkHYPcD7Z
YHod8IRF1//hfsZo9CALopB54Yil8xjfs40n9Xp2F3N4zQSbcsC/BPGJ/LuH1JY6
/Favnd8NaQj39aB/ZB7ZBs4n+Mx7jAFJsZdNQm/GWTRm2YQjEiY7IBEWWBb5AIIf
AMu86ynB+FGY8TBL2SjwswJEnMDkvvQERGM8n07vtj/NvWkwDvhoO/YSAIQv/6N3
w7eRiO0GYrjvaQGc310BEyQ+a9IuNveIp+MoufWSETDSnyK0hlRU3PDMBdqb/gRW
opp8q1fuOI28kQvzd2fRaD7lTWs8tpHy6dhhAgfOBMl16jX4sByLXJhTcgfotWLZ
ZB9OuP+xGcUtDUQzUc0OWytOslUvE1q3JO2G/LaZY9OGP00dZov7Jvf88XaUtuqg
IKX59Wzhjeaz2R20yNHFzD/eToM0E8ugIVnZcFgDJo5fQuDwVxtlWAvGrPno/K6f
3LjnsHj51Tye8n44+pHf3cJ6SpuCNCQIgIePvg6HB9bUGoK/NSZGBeac311mSRDe
EGfgmWhUveST/G61oDnh2TwJ2dnb01OkB+cIlOcF2wJ2bb3MYtebTlFLkHltkB9R
TxCPXhCGFi4qwrH1Uq2QFxK3agc+lNoEU0zzfV3R1TTcbgn+39umJydmNBJVGrmC
nP8+CW67RLfrPizKHF+1SDVXdQ9BoC3drQ5Sen7nnpwdXgyOmzlMLdJp913uncRh
yVhKrixoIVBlQjTR/VT8URgMUS7AiTOyCEYt4NOU01BF2gGulVOUtq0Zn6NglFON
EReqW9IOkj7O/miwmDO5d3LkUrvhmHiOrl3cnWjdjBPOqVvLpnKBtQKLOFL21XSB
vbCoydUWqrx6ATRntsguGxO9UIml5oLO5hTWok/orNjxSuqKuKtMjBG+XPbSeLC1
NVZsBy6fRSEX6qYIZth+NkeXyO2PRjlkDqNhlZIZkLKOGVujtcwaQXDGVjz1Uuid
mMf53RsvjnHSr70UVnzTWo26f6tCg4nChTYOiYJ9WzCZhVEGW/gcXB/hOyg8ijk9
vQhkywpKRhv5n6lk2gGALd+xtn34G4Y1ph79wOBbVdCifqkKGhGU1VBTROokPY0f
eHaS8VmlGKUttOCMTlWoOMyFpiAdGuP3sK9fmXp3GfzK7Veg8jtKXQZJ4l7yTFpR
eP7i07ygPUqE+cctiX32pnOeshmET+yaswwRp+h3Tnl4k03YToOU7SbKIjYi7SD1
BaeQNQPyvFnA/sF24Gtz0x78cAp+fZOAM2KSoFnxyBJpoBdchlxRiok0ovMbhHOe
60DLHdhQ5FEFb4gnTwVZqzGl1uBTPiOHGx32FTj0VHAoxyJcgjWhqUWnV3kVxY1O
a4rDMs0QRPHIMj7XCfc+WhaEhjAG8JGY4tD350nCR03S3Oppi8UnJg5RnheuZ0Sq
MhvjJJrZsQpOEid3QDZE7l22O660GODzFuQNh8B0dARxlm1CZtSaCj+pXvutkdsY
2IItzWFvBlev3Z/6F/2LHy6B8Hfv3jXuHYTPrWq2wFothkcWOrUaft2j9Uw5ah6f
luKlNxRNl6eDrdZ0jOd35Q5f/c/gEKLYaDxOeRaNbePnKAa30CWkmQhPET8PgaI3
IXpKUO1dPggqHQ5HgdLEaabnd0cg1surC+BSM+cGgUGrI8dAy30MCyvC7YYdmB0N
jgcXF4Mjt390dDG4vGzid4u1V4s+BSdzQTbsXSdXzRLaNYFHhHOg8m2YUMOkxAoR
TwqWIc+NloMwtPox8o7dACwUdqMtszniIv6PklaFV+hIGPlIIDlkcYK7VKkZXF4v
y5Lyi3TRC0zFQEhffpHwuKK7l7ohuTmVr1L+ac5DvwIbvJwJt6T8buKlkwrCYOYV
85BzsLbHkCeBDztAH+bnFBgRLeRExRug8Xo+HnM1gnt1fnza/+ESDNNx/+3pFcON
QDe+6l8Orn4+HzhMQI+n3k1KkEX11gOMqiSZJd5nWOQVPPNxs6sQTOBPFkrtFkw5
xmpifZbfB1mV6LA15F+yEvnSoJopyIaKjmSq7I7UUKmilZRde1U80PmpfCtP/QQx
LXpVOQLisvjSxCA3TsDaFSN2rUX4IJWBbIFUtr5Yolpv9BItZHhMF3jIq+0RnzqG
MgzG1J4ALYBDuAseZRgxq2jlFzCsBVnJzBhmAskhFo+jKINdE/4FQwdrBXaBpIld
HLa+va78RXpd8hRlKrUngw9yGAGFP4sRALa17fiuYbutIn+6tL9fADh8ECIqQAwl
RJHIXM6FQn2Z8oV9nX9pYiMwiTgj/fml7qWIAK2ccfABWUQoTHwW5OJ7zAjco8Dq
sFeqZIPaeNnWS9xJ6/juB2gImUdZW2iEvywfyGHX82A6gkbUT4cpM0m+2x3TmdcW
YlLhF7h0KWJCVxJ9pYCn2zRUn5lgIEgBaTYB3doi/9Jh3PMnDPch9DRvJwE+pYwH
2QSsAvrHDFjkIZ6UA8kC7IAQ46ep0s7CPWCPaUZuNHZVZm2TNR414F9qD0LVrhAw
tFsJT1ONQDWgnn8E0fVYm5yAeaje8JGGHnmZ5+IW6kho2k5hKnZv6qVBxgG47VUg
SwGElXAIQFoMAMF+woXF4yFqKs4vSOOPN456AujrKJqCS5XMSd0xVS9FSMKRWXSQ
fKJRoBm2Po9Rv8HfColqseQcVjzMUNAtktUVDENyJ1mn6pDAaAbJleEGImWfos0C
HUyNqO981wqnYGHmH80TKSY6/DDBW74OSJN5GKIKByGL4iyYAcspBAAh5EYooPQX
j4BAZ1GG0/IwXRSB2QxmHAUxi5mPoRoNmFKsdICkeGwDDRAJFF9QugLxEApoiKaw
zvEpJO5gb0f/5WOPNAvAB0FPn4+261WnFPXqY4r6b4UjmGLKwSQbjk9OB2xjHMOf
0wimYFaInwCYpfKWKueUFLpAE7JD4ZhwoDjRHXDFSkD5Bh9U58BhoKEi7IRBgUuZ
zIGAJjpCo3tye8Fl/f5N/915/+r16eAMFvreB/UOR1n0biN21NZEhjZLYDxjKl8y
A7cs1TAER2k8jW7tdANaUrKrlCQL0CpGDCfWsPNZJrGG+018J/dEk1SF3SEGoYyj
2BAGG1Jy3VCbJr63M80LIuChin61L3DARCCMqMnwo/QWUAcC4x9hKIdtdbsO67Z6
6CFcQqMwPDwc4WomnUZ/QiX86IzVvQDpnsLUj6HrMXQBRJSraH9p77f3dl/ttQXp
gPKVNzpgA4HtEFXSm7KjAAxBcBN6QJRAXxv70yglPL0Vg36coJo3Zgm9B6ZMWS9K
5RzB9uee8dumldZaktCyWLVPnKrXcnpPOJdxpkcJzUtlyi0ucAzL7ogDMh3zgjVR
p1pSc8npQuXG1SJOioV30evJvLAmrrAW2y1xgKuHJVGKToLl01UIJ/5MpWy77Q7J
1uRytIRt0SI4e22NtIBIWLmdtjxNqylzVCLqcgIiLlJFNnmlnmi0VusJZnAljmhL
uVJvbUxX6q0t6GokW1YWAHb3gaMGw2adcsMPoFmxW2+pGHe7SozWhrGyehmSc/YZ
tczmh3mFq7IWWwcVRYdbgwnHGzFtxJubuLpwm2jBZuorCjZigkNN7o9GaPhwgw/B
J2bXd5k0T4UJv8jvb9ABnSlxEIgRU+4chl69UAEFEQNjockiR4B8AKAfQ5IeSz+C
EaMhreyxsKSUJ0ZffOZ9BIcETSc6ERASwrqzVzfsTX6UJFwGtJbcctu5ENi3W4Hd
9o62AppGbQhOiZwlJsDWkE3W3bcJ2WlD0x+ksEiR4I1ki06J5ZQUZDnt1UqGklxw
6kyuiLDOmB5df7ROZ/dZEqrtHWiW1hsMmvCCHEtb2ZZ6vcU6LSVNcfqFY7WkRxFS
wGYFXmL8fIgtkFOITebdm956d+BAzn2fc5kuqW0ImPV/t9f1iXMJPpeZpwM3Uihw
UjEEQAyB8FdpkVUrZn4VQpwHlDc3YnligeNukDOz/mSdhhGLDUanIwdYkeqwAVPW
sNXHTFUEiKHIJ5Tb3mJGyCni+1wtBkKIIWrlDIAuPZBHlNR7a6snyMwz0BDSocd7
olqQfcFhrY1ydVEifIvQ5YelKAVK88mfu5EfIk+UpKNDHnzOepSP3OzjIOuYqoDc
fYXB/k94+tNsDIfDhqOOpsp/tDSPJFGX1UQ5zDqu+wcTJkCcHHH0ynqCsMrzFWJa
ViYtDegD5AlP2YQlJiKhYdjCuETGwphCKZ20teolCvUZW/ksr5TdQZzqRKsiphJ+
GekY+tS2Dyu46f7Ek2toPJ56N+I06/Iudf+VQLh8mYFRSpqNx8z2ZD3Ygr6fjmTi
hVzx79N/h+Kkl0azgwnlqmLtXZ0mWDzgKpRrVB15iblUnpojP0ZcicQdz0O/ifUl
hbhTqLhw3YSTjmr9hlpRn5qt3PE4Vj2iGZTVj5ULQaJEPx966dhIgCxw1U0lgDnR
xly9RImwgM/Mp7GoIgWnjWeeqIH5gBvEINIv4I0bTcUnK37GR0tX9aPQ1wLvEu+W
xsG4Fe2GmBs+aTp1cHs9H+cDeQqeE4iG0FNJXXDJR7Kc9MEo89E4tsPL4yiZeVkh
thQbbSHAJM+FQkzaKL9PGzmNrIi7gGzlebGNFp1Bum9ETZLxpTdpbyTSqP/DETCg
geCpEKLnI2EaBAMAcdwiAsN8qFkmFxiKrFviNJFXh71eCBuY4/4Yv3EODjoDdlqF
BiwZCUT09astQdjENNgqKYAFkydciKghY2rk+zFWYwFxi0X1Xo9N7s4v68LRSL1b
sZ8FlFtApoDHStuaBiBDqBSaVrYsE0DHTPzZD0dU20D8MbMk+1CkD7ljsC0tODHR
I4qECCYooBfV1crB0j4sMSjkiAUg3urkLrsNwF0jY6OSH490wfZSkVgpigViMSlk
IrHHGrIjjkaJjM9eMMX6aat2RW1esMsYuyADjqKB1ocv3OaKYl4OnZaTNJiHsGCO
AQeeBlnwWIQ0hWVubNVWZ0/uLHJjMZ1bVTuRgrS2HcH4Svs7D2Uxv+tHI94sVFUz
cx4lnq2iZBwib18Rg21AbbXU1SuKNCroV9G37ETqqjpQIhMTyDPvJvCLGx62idq8
ZktpJirz+/YHtgah03hMXnfTwqCav34VOwt17+S7N033ly8Ze9aqhuoug+rsL4Da
WQbV3dVQizeLvMrT9th4haU4hEUe4oPV+H67226n0tHTPm4pM4kCK8WkQprGjNAe
scmeOUJcW8CUnrIEI77YQ5DFaYfQSdZdYX9TpSjVlRp71fPNVVyJ2aIDJ2+a6Aok
MVtlgDxGdMkqEw25nBOyQWhw5UIZB1/caRByV2QtUqucMY3mic+tA9tP4FPEldov
u0r914nI/Aownaw6x+ALH7nija5uzcRRMMRm/04gyiyLwvS34au3i2asF1K+88Ld
4FPlFG3gVjGHasWpRDLFVxufKAAM13VstNGMyUsRHUPVUYWxNZnClTHkvQx58ABa
49uIe4Xx2uu5yjhCxj6piKKpS8p1kWxuJlRpLQ2WGN3Y17xwKtVH6qzs9A121lKu
vKV1ykpREHJJZS3tW13K2ki4h2IOehdeKnlT0Sp2Z4An7y4I43lWSQLYRAlbMBFF
jVppvX77FaLK20N5vsvoQbIfMGPwABGV3vWUBOUCr4qwV4itNTpzueWh+w5q6WqI
h4qGCwXQVmSu8gQHBRbmSqCNqq6ZSa5Zs1wz07TAchNekzNew9C/glLpMuVCRPta
Sr744EEOa2dtuY+bgkPqTyiXTIroYw2TOKM/oDQELYfCejZKZBijHBltcSSmQ4lK
Pg4LmAse2VLMpV1Y2EkqJBw3G4OLi+GFTGrkN0C5255jx6bOGSlCR3zszafZwUq+
dy5ovQZnBJPwVArdkKdNI/t2lilMtl1a5RQ+5FXUZa209gUHX7iPLobIgAy+WPwS
q9SedtnrWDl1JCYIvMQCmYRrF0RXP5vUUVlienWK3ObaGrPLtdXxLeajXJeqbFyq
AzKFJRRJWbc5oQ9WgZAKqLwEDgQbE7GczvWpbtc6vf9A+XDZTaR38dzTvnigz4iw
xybrspcvSsf4q4ZjuXhM33jJn+QX0oX3drZf3rMx3JR0EaCcnGz6YDLd+RfoRHyw
ssuKcQsDMfuOTOkCqAwzh0P0rPO35Jh1k8Pc0CJ2mjEXpkmpIJL8cBI3BjiUywPp
ExqRJqlMGIv6sobSG327jxWTv++HH6QRQGJa0uVW60G3K6pxyEcy51KKi+0C/nIA
KqlekPLEnCaj8wCMwqtSnKWcpVzq4kuwqKlylTVRWolrM5dDMNkD+/Z3Jz9r6tTL
380qXBYvZ//vvynh7BryYc6gUQeCTmMuzID/ZA2TkDhgjbdn/Z/6J6f9V6eDYv7U
vioxi0t3JHJ3FKouKRRK92fxspp9aYWoYp+uPKL49NsFWWrtGSl2UbEtBph3zfLV
anXEIZTGALlDWWtGTFaW8oLL6mo60MXL+viXqP6u5So+28gRU3Caf9kRL1X96r21
u5C0hKrtNo0ho/siht/yiVgnFp1IKJ//fPV6eOb2z0/cnwYXlyfDM1ScB1Y5uSbU
pZBBGoNKGI4stgG52wtsqSXT/M4PoFclShpbYV2j3v3dv13x38/v/5jff4GvbbD3
/uQPH2P577909/Z29uXvv3Q7+90O/v7Lfmfvv7//8ld8TvCQ9QA2gmweg7Gsv/j9
n/rF4SUFWAfsif85TaIoexLTj8uor1GQZk/SxH+ihnU+18HEJwH/jJU/CXxRlXhn
u9Pt1EfBeMy2fLaV0LOhdWNjQz/Uuk/ZWfSZdUGhWGfnYBf+22dbqGA1gWZra8v0
3mNH3Bed223svNcVnRGp/aFBus/3nZ12h1ED4hENAAIfXc1N9dzptgcrKRw12eBL
xkOcRnPdnURZCv+vO+y9ftj21z+0WGsVcLzpA96jAJcPq4N/wWMoTPikhMA8iv1b
Idp8EJHeSQiP9ZNRihKLmMfMSxLvTuZb0xXIpP6Emf7KEyfVVPxG0RMx7mi6PfmL
9bU0/gLF7W53nll6i4/MIhp1qoSq1t1nx/xaanDnYLdzsLMrNZjwoeKVob5Nkzv7
Trcr9Jix2vmPP7hHJxeDw6vhxc8Othy6r96enF6dnNHT+c/u8cXwlwE8PaKnw+HR
4GJwOXx7cTjAE743ns+icEqHkIyBB0k6IH/tCFlKV7TAP0PacewdtWR+39hOcXDo
cvLm3H09HP64nJAqRdr2/0412vYXKlG308lpETwzTXJJibb9Wve5MYJd0IanB7tt
rUKIraRDAPRtGvSsve886+wYWyga9pVg63T0eyx/84n0ovp3nGQCVmQ0HSZqKlT2
lWo2wN3cXArcLUGbbgqDHr6kDBv2T50oRBYC1EKVolSvMXvoZmq8DZvCpfi7lQNs
mgGs9g0bfZF8d5xEv/JQIhcPErfM1igRnOJlfHPvCMMhdbUt0zVx8jfO/nVy9ZrR
73q8GZxdDY6qJf/8mfO8bUn+ebvjPN/ZU5IvsxN2mlIOXdaYqcNgxVKsD5nykJiO
7JAciN1xjDvLb8grg2QCs5niLXT1l7jCrNKJ0LcsjYSn9EZm5PFvfEyoaionKkO2
IHZND0fkEBaqjODWuT7KMpfbEOgL9On00gU8N+VLgzHf2yQCBfLUnmAu+wLP9/hP
9emijdWMFf9fBpBXakhwghMy0UPiUQmU3Lys2rcyIaoMRMjmXjBt4YKCdxVrarGW
WWsKr54LIeKwJQ3bBB6A5tH1UYeFCI4rqSaVEzXTZFJZlXqNR7HmXtXqed5pO8+7
bWv17Had53t69dQWzXo8cvHWaBaE9Itg+Btg8L/eLO97DwArA2AB5/ZegaDqXtdm
54NZMVipUH6VW7y5GwKbnf9t78h22zaCz/oLykVrq1ZckyItKUYKFKhTCGgOVDHa
9EWQbToRHB0QHaUu4H/vnLuzPBQnMdw8RA9BTC6XO7Nz7VzEeb8DSuNmDhd7L1+/
GE9ejJOOQWwx3eQXE4dVe9Gimzear3MGCSMaXoBBzEtcyrQo8vX1ntto50gXkWjL
0ZRkqvVoKPPvVJFWu8dprztMh86+IsdysFgMReMNjnJ6Cc6LMa7/NqoGy21OuvsS
5R8u9Xm+fmtW35wgQrTXB1E+SB3tfelCPXs1JBHc2+pr0Z4B2o96Du23utsMlV+r
AjQrlKX2XIusuyJcHrRrpiSjCpLuZTt5twawW8OB263Ph+9z9+kLgL4v0qjf+aOs
O+wboRqD2dpFd5Mn7UtWin/OFpOnsNw/8jdghufr/EJCBhKuBvmN5fmXXRFmsiQp
B22HSG6R2kC7YcW68uNg7uyUgIQXeuhEjNXCOADFMcgsjHCSig8T536oFl+R/tKO
B8xWPkdkdWM43d7a0NJ/B/S4LlusR2esrIO8I6eGJ2+XyyuR9dgMF/T96Xj0/LfJ
6fMRntkcGl/enC5m5z7va9Nx6JRF+REnC4py+wu/jHU+eK5BDAwBUcO+RVTSB0Rl
hhiYdkW6ed2h1S9w/Cren52/A12CR5d9WXnbbSit/OSf6fm1LJ82v4yNUk83kwG/
AQVsG1UJYr2YtbO0Q/vM4l+N2ZaabfVxy/KUmMWx44SPTuoTasLRbkitsduq2Lt2
aNg9UcY3WSn4Mv7fk0jHtpy1ogf722N3M+A2gwsnQNyMeue2Ai08pgDKTSILHRZE
9/YjHuRouUQRSAxKyVqZJaPZLgoz+TZGuGhIfT/ywWkwScanT5+O/hqP/j7BYLuI
JGUXY1Ij53TqLU4QgwnKwoAjjmK4NIw9R9RMFgKCZ8kNdZGezhYFxcz5WC8o2TeA
/BylmMaAtiL8+Simxiq7q117MeGLM7xID6Mxrjd7fPPf4ImULx7sOlKirtLYkhI7
P2hbADzxowlI5KTsawPIFKKjEKSJieEZkgQYjdYxWwLMLuhnedc/V+ZZ21Q8TC53
S6yjtpCSPI3ajmmNUsZDGkT/dGoT6ZaBISjmDSHDl5ZUFnWB6AlQvIPFb5u697tB
dgk88dZ3s0IZSxMyV0kmdNAlGpBwZu3z5rBqJN0MU0f3y3wsKprV23x6jnWixVvi
gJ/YzgduG6/y/OL96jE33iluigOyzijFg2o2qO8WbAnIrSZ2HaJaj/3x4Uckc2xd
UsDJZRl9wKJQLGDA3ABMBTpbUkeX6/UNaq5rTKFZXumTVM0K9+cUtQdpRylpReeA
B5DPtOXk0ojWN1q8fDc9z7EhHpsBJ68mo1cnz5wdAIjCx6ylUB5ybBT+s+m5GF38
dmGiak7tRssbsfDEybkmfQHA+D+eiJ/On25fT9RPrI7gZjGnCsNOWTZzPRyYUvbp
sLDkHfa6cXzorfgH2F1hkhrTRa3zz9z/jxBAuwF9XyUZ3M1854KoByOZegGR9FBA
DIw+B4oCqoqNhRtiwAZNHhILW84z8WGKYKQBGD0wS+Je34OBRgHZo5R9Il5lfD0a
BuaHeV7IMoWrW5eaOJ0E02knyys+dVs/npjBgG2q4gdjgjJv55gCkRc6xX2iyjvj
LKLQ6KzHU4bmW2bNtzhFPKV3xNOn4OCBgG2miriXIWj2LJ8kcCnpGWM1OHCoJ6R0
Fr/7YsODeKOTIU6RYNPMxP/jrJd24yxz0rxaaBBF1SgX87hWDwdREPQ8X9PXRJqC
HVuhCR6Yl+MJW6MUrbk2mVa3Oi+Tusxo4IOWpu78Oi+/5n5W3fgGJ/tRTfCujJSa
KI9HjvjpK4Xvx6Wrx+rNz9drE/WxvmM+ulCrAmwWDAdWPWgympoqFT3YLE0x2jlb
F9fcmo5bzqHizneLCFt5uDrtPWqaFy3ATozoXCwyRisLCMIm2Z8lIOizxLqBjtC7
cZQa9vDN+efCjpJWHxGsJN8UyMdyQikFVre7L2wMpaUZxVEU9M4WoUFHQj9jYMxX
/AbNmdduiqrrQmCT0BSC6woU7uJJJmnFy/+V7K6rxfIDnBs+oKmlef1cQsgUQhUI
3190durdT/GwD+JqOPDpFdX9IHklOcXtAPkcccJrTdEWnLA27MS3lIGMb49TleC0
34V/+lvIpHlZPtDqQmKfuoh6XMGiYvRpO1yJ8URSuM3sx2m6nKSPlzjEZsPMxfsz
G6glSWrRhn7dVUf9nE63SU8359+9WJW96OrkQRICyslXSIzP2DSoElOnw6iOD4+6
SRwnxhorQVUBISnD4IPND7X0ug0CskeqOTSxrNpPf1iV7z7/wRC3ha+tGvRVG5fq
bQcIHv0s6sU73MWbZls16E2pkTHexUhcQqM1GKMroFusdJNGqNSWqZC+qHAOx9V0
uX/X1ImAdX4J8hqbmSPru7ZL/P0zl0EhGWmy1XEGW90rOdK+ED9JCUGOFCySmEGr
Sngr8lQiz+scs75WAgVtQc1iqcXI2bvZG1BWiBT2rAFW+BFswMKOG1duTSdS9HuI
Sj5w/i0hyPBbNeUuM8H3atQtVpPVXnkAdjP0495+PaTyf2f3fvt9+23//QdKoPcA
AHgAAA==
--643623-2675965952-3248042441=:9000--


From bac@OCF.Berkeley.EDU  Thu Dec  5 01:55:23 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Wed, 4 Dec 2002 17:55:23 -0800 (PST)
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <200212040844.gB48iLx06213@pcp02138704pcs.reston01.va.comcast.net>
References: <Pine.SOL.4.44.0211292000030.11583-100000@death.OCF.Berkeley.EDU>
 <077101c29a13$a9b78cf0$0900a8c0@spiff> <200212021536.gB2Far428457@pcp02138704pcs.reston01.va.comcast.net>
 <0ad001c29abf$ed602f50$0900a8c0@spiff> <200212031207.gB3C7ri02667@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212031048230.24140-100000@death.OCF.Berkeley.EDU>
 <200212031958.gB3JwVo04341@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212031747240.27-100000@death.OCF.Berkeley.EDU>
 <200212040844.gB48iLx06213@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <Pine.SOL.4.50.0212041754120.10138-100000@death.OCF.Berkeley.EDU>

[Guido van Rossum]

> > Do you prefer I do a separate ``dummy_threading`` module or follow
> > Martin's suggestions and just turn ``dummy_thread`` into a ``threading``
> > substitute itself?
>
> Two separate modules.  They have different APIs and it wouldn't be
> wise to export a different API in the dummy case than in the real
> case.  (I'm sure you can figure out why.)
>

I would hope I could.  =)

OK, so as I said in the response to Tim's email, anyone who was actually
going to look at the patch can hold off until I email the list again when
I have ``dummy_threading`` done.

And if someone can delete the old files from the patch I would appreciate
it (almost all of them will have to be regenerated anyway).

-Brett


From tim.one@comcast.net  Thu Dec  5 02:04:26 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 04 Dec 2002 21:04:26 -0500
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <Pine.SOL.4.50.0212041754120.10138-100000@death.OCF.Berkeley.EDU>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEAMDBAB.tim.one@comcast.net>

[Brett Cannon]
> OK, so as I said in the response to Tim's email, anyone who was
> actually going to look at the patch can hold off until I email the
> list again when I have ``dummy_threading`` done.

I intend to review it, but can't promise to get to it in a timely fashion.

> And if someone can delete the old files from the patch I would
> appreciate it (almost all of them will have to be regenerated anyway).

OK, they're deleted.  I would have done it the first time you asked, except
the "almost all" bit (which was there the first time too) seems to imply
that you don't want *all* of the deleted.  So applying wisdom worthy of
Solomon, I just deleted all of them <wink>.



From bac@OCF.Berkeley.EDU  Thu Dec  5 02:12:00 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Wed, 4 Dec 2002 18:12:00 -0800 (PST)
Subject: [Python-Dev] Can someone look at dummy_thread (#622537)?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEAMDBAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAEAMDBAB.tim.one@comcast.net>
Message-ID: <Pine.SOL.4.50.0212041810280.10138-100000@death.OCF.Berkeley.EDU>

[Tim Peters]

> [Brett Cannon]
> I intend to review it, but can't promise to get to it in a timely fashion.
>

Cool.  Thanks, Tim.

> > And if someone can delete the old files from the patch I would
> > appreciate it (almost all of them will have to be regenerated anyway).
>
> OK, they're deleted.  I would have done it the first time you asked, except
> the "almost all" bit (which was there the first time too) seems to imply
> that you don't want *all* of the deleted.  So applying wisdom worthy of
> Solomon, I just deleted all of them <wink>.
>

I only hesitated because I don't see ``dummy_thread`` from changing, but
because I had a previous patch have a lot of issues because old files were
not deleted, I am glad you just went ahead and used your wisdom.

-Brett


From fredrik@pythonware.com  Thu Dec  5 08:28:03 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 09:28:03 +0100
Subject: [Python-Dev] New Import Hook & New Zip Importer
References: <r01050400-1022-F8B266A507EC11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <00d301c29c38$3cfdc1c0$ced241d5@hagrid>

Just van Rossum wrote:

> zipimport.c todo:
>     - rework the zipimporter class as a subclass of str
>     - doco
>     - add a get_source(fullname) method (linecache.py could use that!)

have you tested the following code with uncompressed PY files?

    buf[data_size] = 'Z'; /* saw this in zipfile.py */
    data_size++;

    raw_data = PyString_FromStringAndSize(buf, data_size);

    PyMem_Free(buf);

    if (raw_data == NULL)
        return NULL;

    if (compress == 0) /* data is not compressed */
        return raw_data;

</F>



From fredrik@pythonware.com  Thu Dec  5 08:31:52 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 09:31:52 +0100
Subject: [Python-Dev] New Import Hook & New Zip Importer
References: <r01050400-1022-F8B266A507EC11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <00d901c29c38$c4eca560$ced241d5@hagrid>

just wrote:

> zipimport.c todo:
>     - rework the zipimporter class as a subclass of str
>     - doco
>     - add a get_source(fullname) method (linecache.py could use that!)

unmarshal_code should check that PyString_Size(data) is at least
eight (or perhaps nine) bytes.

</F>



From mal@lemburg.com  Thu Dec  5 09:48:14 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 10:48:14 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-A8FC5EEC07DA11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <3DEF20DE.7010803@lemburg.com>

Just van Rossum wrote:
> Fredrik Lundh wrote:
> 
>>FWIW, I've done lots of stuff that scans sys.path directories for
>>various purposes.  It would probably be a good idea to use a string
>>subclass for the zip importer, but I don't think we need to make
>>stringness an absolute requirement.
> 
> +1
> 
>>Let's make it *really* easy to use zip archives, and reasonably easy
>>to use other custom importers.
> 
> +1

Why not leave sys.path as list of ordinary strings and add
all the extra support to the existing import mechanism via
hooks at the right places ?!

E.g. such a hook may scan sys.path for files which end
in '.zip' and then redirect the lookups into their ZIP
archive directory.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From Paul.Moore@atosorigin.com  Thu Dec  5 09:07:41 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Thu, 5 Dec 2002 09:07:41 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861996F@UKDCX001.uk.int.atosorigin.com>

From: Just van Rossum [mailto:just@letterror.com]
> Gordon McMillan wrote:

> > Code like this:
> >  for p in sys.path:
> >    x =3D os.path.join(p, ...)
> >    ....
> > is very common (I patched linecache.py for this after imputil
> > went into the std lib). Since PYTHONPATH can consist only of
> > strings, it seems wise to tackle the issue (dealing with strings
> > that describe non-directory collections of modules) instead of
> > postponing it. Also seems sensible to make it so that if X works
> > on PYTHONPATH, sys.path.append(X) should work, too.
>
> Erm, I'm not sure what you're saying... Are you saying that we
> should fix all cases where non-strings on sys.path cause problems,
> or are you saying that there's so much code out there assuming
> sys.path contains strings, and that we therefore should stick with
> strings?

I read Gordon's comments as the latter, which implies your approach
(A) below.

> Both positions can be defended, and both have their problems.
>
> A) Stick with strings. Hooks can be implemented by subclassing str.
> This is great for hooks written in Python, but subclassing str
> in C is not straightforward. Things can still break, though: eg.
> os.path.basename(strsubinst) will return a regular string, not an
> instance of the subclass; might be an issue.
>
> B) Allow arbitrary objects on sys.path. Hooks are then easier to
> write (in C), but some code breakage will occur. The std library we
> can fix (if needed), but third-party code might break.
>
> I would very much prefer B, but if it turns out that we can't break
> the string assumption, I'd still be happy with A (rather that than
> nothing!).

As I say above, I prefer (A) of these two. But in practice, I don't
see the problem with Gordon's metapath approach. Equivalently, every
element of sys.path must be a string, and there is a dictionary mapping
sys.path elements to Owner instances (if it helps, you can say that if
the dictionary doesn't contain a particular element as a key, it can be
treated as a normal directory). The advantage is that we stick to pure
strings, not string subclasses...

[BTW, at present, sys.path does not seem to support Unicode strings.
This seems like a minor wart. If you allow anything more than strings
on sys.path, I'd suggest that this be tidied up, too...]

> Regarding PYTHONPATH and sys.path.append("/path/to/my/archive.zip"):
> for now I'd suggest that the sys.path traversing code checks
> for a .zip extension, and replace the item with a zipimporter
> instance. This check can be very cheap. Later we could add a general
> extension-checking feature, where one could register an import hook
> for a specific extension. This might be a case of YAGNI, though...

Is this check going to happen whenever sys.path gets changed? If so,
how do you trap that?

And yes, I very definitely need a way of registering a user-defined
hook for path entries (and not always based on extension!!!) Imagining
a hook to handle something like "http://my.repository/python/" on
sys.path is not hard, or unreasonable... (Security considerations =
aside).

Paul.


From guido@python.org  Thu Dec  5 09:11:17 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 05 Dec 2002 04:11:17 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Wed, 04 Dec 2002 15:49:34 PST."
 <3DEE948E.3010802@ActiveState.com>
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico>
 <3DEE948E.3010802@ActiveState.com>
Message-ID: <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net>

> > I would it support this kind of code:
> >
> > sys.path = L
> > L.append(...) ?
> 
> Sure -- sys.path wouldn't be a "real" list object, just one that
> behaved as one.

Note that I've seen people write

  sys.path = ...

where ... is a list they calculated (from the old sys.path, to be
sure).  E.g.

  sys.path = filter(os.path.exists, sys.path) # remove bogus path elements

I still think I'd like sys.path to be a real list of real strings (not
necessarily representing files or directories though), with a
configurable set of handlers that are given a chance to deal with each
string in turn.  (Or was this already rejected?  I admit the Zope3
sprint is taking 150% if my attention this week.)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Paul.Moore@atosorigin.com  Thu Dec  5 09:21:29 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Thu, 5 Dec 2002 09:21:29 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619970@UKDCX001.uk.int.atosorigin.com>

From: Guido van Rossum [mailto:guido@python.org]
> I still think I'd like sys.path to be a real list of real strings
> (not necessarily representing files or directories though), with
> a configurable set of handlers that are given a chance to deal
> with each string in turn.

That's exactly what I am arguing for.

Jim Ahlstrom's existing patch makes this happen, for the extremely
specific case of zip files only.

Gordon McMillan's iu.py code offers this in a general way, but in
Python - and hence with a (fairly small!) efficiency hit, and no
obvious way of writing hooks in C.

I believe that taking Gordon's design and implementing it in C,
along with a standardised zip import hook, satisfies both Just's
requirements and the requirements of PEP 273. But if it's not
possible to do this in the available timescale, we should just go
with Jim's patch for now (and maybe add iu.py to the standard
library, as an interim pure-python solution), and look at import
hooks in the longer term. I see no obvious value in going through
the design process again, from scratch.

Paul.


From guido@python.org  Thu Dec  5 09:27:02 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 05 Dec 2002 04:27:02 -0500
Subject: [Python-Dev] Mac OSX issues
In-Reply-To: Your message of "Tue, 03 Dec 2002 08:20:07 EST."
 <15852.44935.756518.645276@gargle.gargle.HOWL>
References: <200211280416.gAS4G2221621@pcp02138704pcs.reston01.va.comcast.net> <15850.58353.356475.182137@gargle.gargle.HOWL> <200212021001.gB2A1CZ24699@pcp02138704pcs.reston01.va.comcast.net> <15851.43445.443010.182362@montanaro.dyndns.org> <200212030844.gB38iEJ01481@pcp02138704pcs.reston01.va.comcast.net>
 <15852.44935.756518.645276@gargle.gargle.HOWL>
Message-ID: <200212050927.gB59R3A11368@pcp02138704pcs.reston01.va.comcast.net>

> BTW, test_socket.py passes just fine for me.  hostname(1) returns
> "freewill.local." (yes, trailing dot).
> 
> The difference may be that I used Netinfo Manager to add hostname
> entries for all my local machines, including the MacOSX machine I ran
> the tests on.  My Netinfo-fu is pretty meager so I don't know if I did
> it completely right, but it seems to work for me.

That fixed it for me too.

I still think the test should be skipped rather than fail when
gethostbyname() raises an error.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Thu Dec  5 10:25:01 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 11:25:01 +0100
Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies
References: <BBA757DD-07DF-11D7-B2E7-000393877AE4@codefab.com>
Message-ID: <3DEF297D.8080603@lemburg.com>

Bill Bumgarner wrote:
> 
> I'm writing a Transport class to handle proxy servers such that the 
> xmlrpclib module will be able to communicate with a server that is on 
> the other side of a proxy somewhere.

Doesn't urllib already support this out of the box ? You can use
its machinery for finding out the proxy settings and then add them
to Transport as needed.

> Most of the difficulty in doing so is the need to navigate the urllib 
> vs. urllib2 vs. httplib maze.   I believe I can do what I need entirely 
> via urllib, but may use urllib2 because the capabilities better fit the 
> needs of my ultimate use.
 >
> However, I would like to create a Transport class/subclass that can be 
> dropped into the existing xmlrpclib module in the core such that this 
> problem is solved once and for all.
> 
> As such, I would really like some feedback from the python-dev community 
> as to ensure that the efforts I put forth to solve  my particular needs 
> are applicable outside of those needs...

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From just@letterror.com  Thu Dec  5 09:48:13 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 10:48:13 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996F@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-AF98707E083611D7AD06003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> > A) Stick with strings. Hooks can be implemented by subclassing str.
> > This is great for hooks written in Python, but subclassing str
> > in C is not straightforward. Things can still break, though: eg.
> > os.path.basename(strsubinst) will return a regular string, not an
> > instance of the subclass; might be an issue.
> >
> > B) Allow arbitrary objects on sys.path. Hooks are then easier to
> > write (in C), but some code breakage will occur. The std library we
> > can fix (if needed), but third-party code might break.
> >
> > I would very much prefer B, but if it turns out that we can't break
> > the string assumption, I'd still be happy with A (rather that than
> > nothing!).
> 
> As I say above, I prefer (A) of these two. But in practice, I don't
> see the problem with Gordon's metapath approach. Equivalently, every
> element of sys.path must be a string, and there is a dictionary mapping
> sys.path elements to Owner instances (if it helps, you can say that if
> the dictionary doesn't contain a particular element as a key, it can be
> treated as a normal directory). The advantage is that we stick to pure
> strings, not string subclasses...

What's the advantage of banning string subclasses? What's so bad about lifting
limitations?

> > Regarding PYTHONPATH and sys.path.append("/path/to/my/archive.zip"):
> > for now I'd suggest that the sys.path traversing code checks
> > for a .zip extension, and replace the item with a zipimporter
> > instance. This check can be very cheap. Later we could add a general
> > extension-checking feature, where one could register an import hook
> > for a specific extension. This might be a case of YAGNI, though...
> 
> Is this check going to happen whenever sys.path gets changed? If so,
> how do you trap that?

It's trapped every single time sys.path is traversed. It's a very cheap test.

> And yes, I very definitely need a way of registering a user-defined
> hook for path entries (and not always based on extension!!!) Imagining
> a hook to handle something like "http://my.repository/python/" on
> sys.path is not hard, or unreasonable... (Security considerations aside).

That would be way cool! But let's please not try to do too much *now*; let's get
zipimport going and see from there.

Just


From just@letterror.com  Thu Dec  5 09:31:13 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 10:31:13 +0100
Subject: [Python-Dev] New Import Hook & New Zip Importer
In-Reply-To: <00d301c29c38$3cfdc1c0$ced241d5@hagrid>
Message-ID: <r01050400-1022-B2FB4836083611D7AD06003065D5E7E4@[10.0.0.23]>

Fredrik Lundh wrote:

> have you tested the following code with uncompressed PY files?
> 
>     buf[data_size] = 'Z'; /* saw this in zipfile.py */
>     data_size++;

No, whoops! The "zipped-uncompressed.zip" file that comes with the other patch
is actually compressed... I'll test & fix.

> unmarshal_code should check that PyString_Size(data) is at least
> eight (or perhaps nine) bytes.

Ok, done.

Thank you!

Just


From just@letterror.com  Thu Dec  5 09:41:20 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 10:41:20 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEF20DE.7010803@lemburg.com>
Message-ID: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>

M.-A. Lemburg wrote:

> Why not leave sys.path as list of ordinary strings and add
> all the extra support to the existing import mechanism via
> hooks at the right places ?!
> 
> E.g. such a hook may scan sys.path for files which end
> in '.zip' and then redirect the lookups into their ZIP
> archive directory.

Then you get the problem that Samuele pointed out: if the *hook* scans sys.path
you get a funny path resolution order: hooks[0] may find the module on
sys.path[3] before hooks[2] finds it on sys.path[2].

We should document that

- sys.path *may* contain arbitrary objects, and that sys.path traversing
  code should be careful.
- the std library promises to not add objects to sys.path that are not
  isinstance(p, (str, unicode)).

I think this is the perfect compromise. Thank you </F>!

Just


From mal@lemburg.com  Thu Dec  5 10:51:52 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 11:51:52 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <3DEF2FC8.4060700@lemburg.com>

Just van Rossum wrote:
> M.-A. Lemburg wrote:
> 
> 
>>Why not leave sys.path as list of ordinary strings and add
>>all the extra support to the existing import mechanism via
>>hooks at the right places ?!
>>
>>E.g. such a hook may scan sys.path for files which end
>>in '.zip' and then redirect the lookups into their ZIP
>>archive directory.
> 
> 
> Then you get the problem that Samuele pointed out: if the *hook* scans sys.path
> you get a funny path resolution order: hooks[0] may find the module on
> sys.path[3] before hooks[2] finds it on sys.path[2].

Uhm, I was unclear here I guess... I meant that the strings
in sys.path should trigger a hook which then takes care of
managing the lookup and possibly the import. Much like what
urllib does with URLs.

Here's a sketch:

1. User programs register import hooks based on REs which are
    used to match the entries in sys.path, e.g. ".*\.zip" for
    ZIP importers (caching could help in improving the mapping
    performance).

2. When Python sees an import request, it scans sys.path and
    creates hook objects for each entry which it then calls
    to say "go look and check whether you have module X" until
    one of the hooks succeeds.

3. Python then uses the hook object to complete the import
    in much a similar way as e.g. SAX parsers call out to
    event handlers.

The idea is to reuse as much of the existing import machinery
as possible -- writing these hooks in C wouldn't be too hard
either.

> We should document that
> 
> - sys.path *may* contain arbitrary objects, and that sys.path traversing
>   code should be careful.
> - the std library promises to not add objects to sys.path that are not
>   isinstance(p, (str, unicode)).
> 
> I think this is the perfect compromise. Thank you </F>!

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From just@letterror.com  Thu Dec  5 10:18:57 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 11:18:57 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEF2FC8.4060700@lemburg.com>
Message-ID: <r01050400-1022-FB57DB79083A11D7AD06003065D5E7E4@[10.0.0.23]>

M.-A. Lemburg wrote:

> Here's a sketch:
> 
> 1. User programs register import hooks based on REs which are
>     used to match the entries in sys.path, e.g. ".*\.zip" for
>     ZIP importers (caching could help in improving the mapping
>     performance).
> 
> 2. When Python sees an import request, it scans sys.path and
>     creates hook objects for each entry which it then calls
>     to say "go look and check whether you have module X" until
>     one of the hooks succeeds.
> 
> 3. Python then uses the hook object to complete the import
>     in much a similar way as e.g. SAX parsers call out to
>     event handlers.

Nice; the hooks can then be cached in a dict, as in iu.py, with the path entry
as the key.

This makes bootstrapping a bit harder, though, as now we also need re/sre/_sre
to be available before hooked imports can work...

> The idea is to reuse as much of the existing import machinery
> as possible -- writing these hooks in C wouldn't be too hard
> either.

That's exactly what my patch already does: it leaves most of import.c in tact,
it adds no duplication.

I'd argue that *implementation*-wise it's simpler to just allow the sys.path
entry to handle the request. I also don't see a problem with it design-wise,
apart from b/w compatibility issues (which I think are non-issues if we use str
subclasses).

Are people against of the whole *idea* of having non-strings on sys.path, or is
it "only" a b/c compatibility concern?

Just


From fredrik@pythonware.com  Thu Dec  5 10:40:52 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 11:40:52 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]> <3DEF2FC8.4060700@lemburg.com>
Message-ID: <165401c29c4a$c9b91c60$0900a8c0@spiff>

mal wrote:

> > Why not leave sys.path as list of ordinary strings and add
> > all the extra support to the existing import mechanism via
> > hooks at the right places ?!

since everybody knows that sys.path is used to specify the search
path for modules, sys.path is the right place.

> Here's a sketch:
>=20
> 1. User programs register import hooks based on REs which are
>     used to match the entries in sys.path, e.g. ".*\.zip" for
>     ZIP importers (caching could help in improving the mapping
>     performance).

a solution that won't let you wrap up the standard library isn't much
of a solution (portions of both RE engines are written in Python)

I think this is overkill.  let's keep it simple and grokable.

</F>



From fredrik@pythonware.com  Thu Dec  5 10:31:24 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 11:31:24 +0100
Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies
References: <BBA757DD-07DF-11D7-B2E7-000393877AE4@codefab.com> <3DEF297D.8080603@lemburg.com>
Message-ID: <165301c29c4a$c9b48880$0900a8c0@spiff>

mal wrote:

> Doesn't urllib already support this out of the box ? You can use
> its machinery for finding out the proxy settings and then add them
> to Transport as needed.

urllib doesn't allow you to override content-types and other things
that you need to talk proper XML-RPC, and my "request body" [1]
patch was rejected by GvR.

I'm pretty sure you can use urllib2 for this purpose, though, but I
haven't looked deeper into that (mostly because people has been
promising me proxy patches for years...)

(Bill, note that Python CVS isn't the "master repository" for xmlrpclib;
if you come up with a clean and nice urllib2-based proxy transport,
you can just mail it to me)

</F>

1) http://effbot.org/zone/python-urllib-requestbody.htm



From Paul.Moore@atosorigin.com  Thu Dec  5 11:03:53 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Thu, 5 Dec 2002 11:03:53 -0000
Subject: [Python-Dev] New Import Hook & New Zip Importer
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E5E@UKDCX001.uk.int.atosorigin.com>

On Thu,  5 Dec 2002 02:00:23 +0100, in  you wrote:

>Phew, it works.
>
>The attached tgz file contains:
>    zipper.patch: patches setup.py, Python/import.c, Python/importdl.h
>    zipimport.c: zipimport module, move to Modules/
>
>Do I need to mention it's a rough patch <wink>?

How do I build on Windows? I don't know what's needed to build the
zipimport module... (Sorry, I'm not being awkward here, I have no
idea how to set up the necessary Visual Studio stuff so that I can
build it - and someone will need to, before the patch is complete).

>import.c changes:
>    - Allow arbitrary objects on sys.path.

I still have reservations about this. Code will break... (if you
mean "string subclasses", then I feel slightly better about it)

>    - If a path entry ends in ".zip", replace that entry with
>      a zipimporter instance. (obviously, the zipimport module
>      must be available for this to work.)

This seems like a gross hack. How do I write a similar hook for
strings ending in ".tar"? This isn't a YAGNI, it's a design
constraint which I don't believe is reasonable.

>    - The search algo is:
>        [handle builtin modules and frozen]
>        for p on sys.path:
>            if hasattr(p, "find_module"):
>                handler =3D p.find_module(name)
>                if handler is None:
>                    continue
>            if isinstance(p, str):
>                [do builtin import]

Who handles checking for pyc/pyo, looking at timestamps,
saving compiled source (if possible) etc?

Note that Jim Ahlstrom's patch modified the search order
so that *both* of pyc and pyo were considered. (This is to
allow python -O runs to get compiled modules from a zipfile
when only .pyc were available in the zipfile. There was a
discussion at the time about why this was good - I can't
recall the details just now). Your hooks don't seem to
support modified search orders like this.

>import.c todo/notes:
>    - clean up, refactor, debug, document

Too right! I don't see any benefits in terms of code
readability for this patch (compared to Jim's code). In
both cases, import.c is still pretty incomprehensible :-)

That's about all the comments I can make, without building
the code and trying it out.

As a point of reference, could you possibly try the benchmark
code I sent you? If for no other reason than to see if it
runs under your patch...

Paul.


From mal@lemburg.com  Thu Dec  5 11:15:36 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 12:15:36 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>	<3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>
Message-ID: <3DEF3558.8010006@lemburg.com>

Fredrik Lundh wrote:
>>Here's a sketch:
>>
>>1. User programs register import hooks based on REs which are
>>    used to match the entries in sys.path, e.g. ".*\.zip" for
>>    ZIP importers (caching could help in improving the mapping
>>    performance).
> 
> a solution that won't let you wrap up the standard library isn't much
> of a solution (portions of both RE engines are written in Python)

As I said, this was a sketch. Things like zip imports should
be built-in to avoid bootstrapping problems like the one
you mention.

> I think this is overkill.  let's keep it simple and grokable.

Hmm, that's what I was after :-) Nothing complicated like importers
which pretend to be strings or one of the current white magic
solutions in use today.

You should basically be able to say

	sys.path.append('/usr/local/lib/python2.3/std.zip')

and have the Python std lib at your fingertips. From a user
POV I don't want to bother about how Python manages to get
at the stuff inside the ZIP archive.

Also note that any solution using special objects in sys.path
is not compatible with the PYTHONPATH environment variable
(or registry setting). The above approach is...

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From Paolo Invernizzi <paoloinvernizzi@dmsware.com>  Thu Dec  5 11:21:11 2002
From: Paolo Invernizzi <paoloinvernizzi@dmsware.com> (Paolo Invernizzi)
Date: Thu, 5 Dec 2002 12:21:11 +0100
Subject: Re[2]: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <165401c29c4a$c9b91c60$0900a8c0@spiff>
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>
 <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>
Message-ID: <013394900.20021205122111@dmsware.com>

Hello Fredrik,

Thursday, December 5, 2002, 11:40:52 AM, you wrote:

FL> a solution that won't let you wrap up the standard library isn't much
FL> of a solution (portions of both RE engines are written in Python)

FL> I think this is overkill.  let's keep it simple and grokable.

try / except ? The first succeed is the winner.
I agree with Paul that Gordon's ui is still the best startup point.

from ui.py

        for klass in self.ownertypes:
            try:
                # this may cause an import, which may cause recursion
                # hence the protection
                owner = klass(path)
            except:
                pass
            else:
                break

And, about sys.path, if not plain string/unicode, what's in package
__path__ ?


-- 
Best regards,
 Paolo                            mailto:paoloinvernizzi@dmsware.com



From mal@lemburg.com  Thu Dec  5 11:26:36 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 12:26:36 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-FB57DB79083A11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <3DEF37EC.4030902@lemburg.com>

Just van Rossum wrote:
> M.-A. Lemburg wrote:
> 
> 
>>Here's a sketch:
>>
>>1. User programs register import hooks based on REs which are
>>    used to match the entries in sys.path, e.g. ".*\.zip" for
>>    ZIP importers (caching could help in improving the mapping
>>    performance).
>>
>>2. When Python sees an import request, it scans sys.path and
>>    creates hook objects for each entry which it then calls
>>    to say "go look and check whether you have module X" until
>>    one of the hooks succeeds.
>>
>>3. Python then uses the hook object to complete the import
>>    in much a similar way as e.g. SAX parsers call out to
>>    event handlers.
> 
> 
> Nice; the hooks can then be cached in a dict, as in iu.py, with the path entry
> as the key.

Right.

> This makes bootstrapping a bit harder, though, as now we also need re/sre/_sre
> to be available before hooked imports can work...

Right again. We need to provide a basic set of built-in
solutions which work without requiring any .py modules being
loaded... or maybe trim down the registration logic to just
look for suffixes in sys.path, e.g. '.zip' is for the ZIP importer
hook, '.exe' for the resource importer hook, etc.

>>The idea is to reuse as much of the existing import machinery
>>as possible -- writing these hooks in C wouldn't be too hard
>>either.
> 
> That's exactly what my patch already does: it leaves most of import.c in tact,
> it adds no duplication.

That's good :-). I haven't followed the thread too closely, but since
this debate has been going on for years, I thought, I just drop
in a few lines ;-)

> I'd argue that *implementation*-wise it's simpler to just allow the sys.path
> entry to handle the request. I also don't see a problem with it design-wise,
> apart from b/w compatibility issues (which I think are non-issues if we use str
> subclasses).
> 
> Are people against of the whole *idea* of having non-strings on sys.path, or is
> it "only" a b/c compatibility concern?

Since sys.path is used by quite a few applications directly,
the backward compatibility argument is a strong one. I also
think that adding too much magic (like having importers being
subclasses of str) will only result in people not using the
new techniques.

 From a user POV, I would like to add the path to a ZIP archive
to sys.path and be done with it. From a software vendor POV,
I may also be interested in adding support for signed/encrypted PYC
files within those ZIP archives, so appropriate hooks to
be able to add that support would be nice as well.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Thu Dec  5 11:34:13 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 05 Dec 2002 12:34:13 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEF3558.8010006@lemburg.com>
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>
 <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>
 <3DEF3558.8010006@lemburg.com>
Message-ID: <m3bs401yyy.fsf@mira.informatik.hu-berlin.de>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> You should basically be able to say
> 
> 	sys.path.append('/usr/local/lib/python2.3/std.zip')
> 
> and have the Python std lib at your fingertips. From a user
> POV I don't want to bother about how Python manages to get
> at the stuff inside the ZIP archive.
> 
> Also note that any solution using special objects in sys.path
> is not compatible with the PYTHONPATH environment variable
> (or registry setting). The above approach is...

It appears you haven't studied Just's patch at all. It allows you to
do the sys.path.append, put the zipfile into PYTHONPATH, and *still*
uses special objects on sys.path for zipimports.

Regards,
Martin



From mal@lemburg.com  Thu Dec  5 11:40:51 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 12:40:51 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>	<3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>	<3DEF3558.8010006@lemburg.com> <m3bs401yyy.fsf@mira.informatik.hu-berlin.de>
Message-ID: <3DEF3B43.4080108@lemburg.com>

Martin v. L=F6wis wrote:
> "M.-A. Lemburg" <mal@lemburg.com> writes:
>=20
>=20
>>You should basically be able to say
>>
>>	sys.path.append('/usr/local/lib/python2.3/std.zip')
>>
>>and have the Python std lib at your fingertips. From a user
>>POV I don't want to bother about how Python manages to get
>>at the stuff inside the ZIP archive.
>>
>>Also note that any solution using special objects in sys.path
>>is not compatible with the PYTHONPATH environment variable
>>(or registry setting). The above approach is...
>=20
>=20
> It appears you haven't studied Just's patch at all.=20

That's right; I didn't have time to take a look.

> It allows you to
> do the sys.path.append, put the zipfile into PYTHONPATH,=20

Nice and thanks for the clarification.

> and *still* uses special objects on sys.path for zipimports.

But why store the import objects in sys.path and not in
some auxilliary dictionary instead ?

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Thu Dec  5 11:44:43 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 05 Dec 2002 12:44:43 +0100
Subject: Re[2]: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <013394900.20021205122111@dmsware.com>
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>
 <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>
 <013394900.20021205122111@dmsware.com>
Message-ID: <m37keo1yhg.fsf@mira.informatik.hu-berlin.de>

Paolo Invernizzi <paoloinvernizzi@dmsware.com> writes:

> try / except ? The first succeed is the winner.
> I agree with Paul that Gordon's ui is still the best startup point.

Can we please focus on the patch at hand? What specific concerns do
you have about Just's patch?

I'm not interested in advantages and flaws of infrastructures that one
could build, if I need to find out whether a specific infrastructure
implemented for the problem has any flaws.

Regards,
Martin


From fredrik@pythonware.com  Thu Dec  5 11:46:23 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 12:46:23 +0100
Subject: [Python-Dev] New Import Hook & New Zip Importer
References: <16E1010E4581B049ABC51D4975CEDB885E2E5E@UKDCX001.uk.int.atosorigin.com>
Message-ID: <171401c29c53$f582bdc0$0900a8c0@spiff>

Paul Moore wrote:

> >    - If a path entry ends in ".zip", replace that entry with
> >      a zipimporter instance. (obviously, the zipimport module
> >      must be available for this to work.)
>=20
> This seems like a gross hack. How do I write a similar hook for
> strings ending in ".tar"? This isn't a YAGNI, it's a design
> constraint which I don't believe is reasonable.

    "Let's make it *really* easy to use zip archives, and reasonably =
easy
    to use other custom importers."

really easy:

    import mymodule

reasonably easy:   =20

    import sys

    # manipulate path as necessary

    for index, path in enumerate(sys.path):
        if isinstance(path, string) and path.endswith(".tar"):
            sys.path[index] =3D tarimporter(path)

    import mymodule, ...

if you think otherwise, please provide use cases.  the implied "there =
will be
a separate category of people who provide hooks and repackage existing
programs without having access to source code for any part of the =
program
or the standard library" case is totally bogus, and so is the "there =
will be
programs that add importers to the path without knowing what they add"
case.

> >import.c changes:
> >    - Allow arbitrary objects on sys.path.
>=20
> I still have reservations about this. Code will break... (if you
> mean "string subclasses", then I feel slightly better about it)

that's not much of an argument: fragile code will only break if *you* =
add
non-string objects to the path in *your* programs.

if you don't want your programs to break, don't break them (or fix the
breakage).

</F>




From guido@python.org  Thu Dec  5 11:46:36 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 05 Dec 2002 06:46:36 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Thu, 05 Dec 2002 09:21:29 GMT."
 <16E1010E4581B049ABC51D4975CEDB88619970@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619970@UKDCX001.uk.int.atosorigin.com>
Message-ID: <200212051146.gB5Bkaq11808@pcp02138704pcs.reston01.va.comcast.net>

> From: Guido van Rossum [mailto:guido@python.org]
> > I still think I'd like sys.path to be a real list of real strings
> > (not necessarily representing files or directories though), with
> > a configurable set of handlers that are given a chance to deal
> > with each string in turn.

[Paul Moore]
> That's exactly what I am arguing for.
> 
> Jim Ahlstrom's existing patch makes this happen, for the extremely
> specific case of zip files only.
> 
> Gordon McMillan's iu.py code offers this in a general way, but in
> Python - and hence with a (fairly small!) efficiency hit, and no
> obvious way of writing hooks in C.
> 
> I believe that taking Gordon's design and implementing it in C,
> along with a standardised zip import hook, satisfies both Just's
> requirements and the requirements of PEP 273. But if it's not
> possible to do this in the available timescale, we should just go
> with Jim's patch for now (and maybe add iu.py to the standard
> library, as an interim pure-python solution), and look at import
> hooks in the longer term. I see no obvious value in going through
> the design process again, from scratch.

OK, except for adding iu.py -- we already have ihooks.py and
imputils.py, we don't need another one until we know what we will do
long term.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Thu Dec  5 11:47:16 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 12:47:16 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>	<3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>	<3DEF3558.8010006@lemburg.com> <m3bs401yyy.fsf@mira.informatik.hu-berlin.de> <3DEF3B43.4080108@lemburg.com>
Message-ID: <171a01c29c54$1060a3f0$0900a8c0@spiff>

mal wrote:

> > and *still* uses special objects on sys.path for zipimports.
>=20
> But why store the import objects in sys.path and not in
> some auxilliary dictionary instead ?

"Things should be as complex as necessary but not more complex."=20

</F>



From martin@v.loewis.de  Thu Dec  5 11:49:57 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 05 Dec 2002 12:49:57 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEF3B43.4080108@lemburg.com>
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>
 <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>
 <3DEF3558.8010006@lemburg.com>
 <m3bs401yyy.fsf@mira.informatik.hu-berlin.de>
 <3DEF3B43.4080108@lemburg.com>
Message-ID: <m33cpc1y8q.fsf@mira.informatik.hu-berlin.de>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> But why store the import objects in sys.path and not in
> some auxilliary dictionary instead ?

What would be the advantage of doing so?

And, if this is done: How can I add my own custom import mechanism to
sys.path? The machinery to replace strings in sys.path is only
available for zip files, all other hooks must add their objects to
sys.path directly.

Regards,
Martin


From fredrik@pythonware.com  Thu Dec  5 11:57:34 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 12:57:34 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico>              <3DEE948E.3010802@ActiveState.com>  <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <177e01c29c55$80b89990$0900a8c0@spiff>

Guido wrote:

> I still think I'd like sys.path to be a real list of real strings (not
> necessarily representing files or directories though), with a
> configurable set of handlers that are given a chance to deal with each
> string in turn.  (Or was this already rejected?

I've rejected it.  Does that count?  ;-)

</F>



From mal@lemburg.com  Thu Dec  5 12:00:25 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 13:00:25 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>	<3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>	<3DEF3558.8010006@lemburg.com>	<m3bs401yyy.fsf@mira.informatik.hu-berlin.de>	<3DEF3B43.4080108@lemburg.com> <m33cpc1y8q.fsf@mira.informatik.hu-berlin.de>
Message-ID: <3DEF3FD9.7050700@lemburg.com>

Martin v. L=F6wis wrote:
> "M.-A. Lemburg" <mal@lemburg.com> writes:
>=20
>=20
>>But why store the import objects in sys.path and not in
>>some auxilliary dictionary instead ?
>=20
>=20
> What would be the advantage of doing so?

Real strings in a real dictionary.

> And, if this is done: How can I add my own custom import mechanism to
> sys.path? The machinery to replace strings in sys.path is only
> available for zip files, all other hooks must add their objects to
> sys.path directly.

Well, that's what I tried to solve in the sketch I posted earlier
(slightly modified to meet Fredrik's requirements):

1. User programs register import hooks based on suffixes which are
    used to match the entries in sys.path, e.g. ".zip" for
    ZIP importers (caching could help in improving the mapping
    performance; this is where the auxilliary dictionary comes into
    play).

2. When Python sees an import request, it scans sys.path and
    creates hook objects for each entry which it then calls
    to say "go look and check whether you have module X" until
    one of the hooks succeeds.

3. Python then uses the hook object to complete the import
    in much a similar way as e.g. SAX parsers call out to
    event handlers.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From mal@lemburg.com  Thu Dec  5 12:01:28 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 13:01:28 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <3DEDF004.25379.630FD638@localhost>	<r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]>	<15854.29224.574966.648231@montanaro.dyndns.org>	<3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico>	<3DEE948E.3010802@ActiveState.com>	<200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff>
Message-ID: <3DEF4018.5020003@lemburg.com>

Fredrik Lundh wrote:
> Guido wrote:
> 
> 
>>I still think I'd like sys.path to be a real list of real strings (not
>>necessarily representing files or directories though), with a
>>configurable set of handlers that are given a chance to deal with each
>>string in turn.  (Or was this already rejected?
> 
> 
> I've rejected it.  Does that count?  ;-)

I'll add my +1 to Guido's plan, if that helps ;-)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Thu Dec  5 12:18:53 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 05 Dec 2002 13:18:53 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <177e01c29c55$80b89990$0900a8c0@spiff>
References: <3DEDF004.25379.630FD638@localhost>
 <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]>
 <15854.29224.574966.648231@montanaro.dyndns.org>
 <3DEE7B87.5090602@ActiveState.com>
 <037001c29be3$ba396400$6d94fea9@newmexico>
 <3DEE948E.3010802@ActiveState.com>
 <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net>
 <177e01c29c55$80b89990$0900a8c0@spiff>
Message-ID: <m3ptsgzmj6.fsf@mira.informatik.hu-berlin.de>

"Fredrik Lundh" <fredrik@pythonware.com> writes:

> I've rejected it.  Does that count?  ;-)

Certainly. The question is: how much :-)

Martin


From martin@v.loewis.de  Thu Dec  5 12:21:45 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 05 Dec 2002 13:21:45 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEF3FD9.7050700@lemburg.com>
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>
 <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>
 <3DEF3558.8010006@lemburg.com>
 <m3bs401yyy.fsf@mira.informatik.hu-berlin.de>
 <3DEF3B43.4080108@lemburg.com>
 <m33cpc1y8q.fsf@mira.informatik.hu-berlin.de>
 <3DEF3FD9.7050700@lemburg.com>
Message-ID: <m3lm34zmee.fsf@mira.informatik.hu-berlin.de>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> Well, that's what I tried to solve in the sketch I posted earlier
> (slightly modified to meet Fredrik's requirements):
> 
> 1. User programs register import hooks based on suffixes which are
>     used to match the entries in sys.path, e.g. ".zip" for
>     ZIP importers (caching could help in improving the mapping
>     performance; this is where the auxilliary dictionary comes into
>     play).

I don't like this idea. How could I then make a HTTP importer, where
no suffix is available?

Regards,
Martin


From guido@python.org  Thu Dec  5 12:31:10 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 05 Dec 2002 07:31:10 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Thu, 05 Dec 2002 12:57:34 +0100."
 <177e01c29c55$80b89990$0900a8c0@spiff>
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net>
 <177e01c29c55$80b89990$0900a8c0@spiff>
Message-ID: <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net>

> Guido wrote:
> 
> > I still think I'd like sys.path to be a real list of real strings (not
> > necessarily representing files or directories though), with a
> > configurable set of handlers that are given a chance to deal with each
> > string in turn.  (Or was this already rejected?
> 
> I've rejected it.  Does that count?  ;-)
> 
> </F>

Why don't you care about the backwards incompatibilities?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Thu Dec  5 12:34:38 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 13:34:38 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <m33cpc1y8q.fsf@mira.informatik.hu-berlin.de>
Message-ID: <r01050400-1022-F5837BD6084D11D7AD06003065D5E7E4@[10.0.0.23]>

Martin v. L=F6wis wrote:

> "M.-A. Lemburg" <mal@lemburg.com> writes:
>=20
> > But why store the import objects in sys.path and not in
> > some auxilliary dictionary instead ?
>=20
> What would be the advantage of doing so?
>=20
> And, if this is done: How can I add my own custom import mechanism to
> sys.path? The machinery to replace strings in sys.path is only
> available for zip files, all other hooks must add their objects to
> sys.path directly.

+1

With MAL's idea you'd need to write

    p =3D ...
    sys.importers[p] =3D myimporter(p)
    sys.path.append(p)

instead of

    sys.path.append(myimporter(p))

Just


From just@letterror.com  Thu Dec  5 12:31:36 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 13:31:36 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <177e01c29c55$80b89990$0900a8c0@spiff>
Message-ID: <r01050400-1022-F5783BB0084D11D7AD06003065D5E7E4@[10.0.0.23]>

Fredrik Lundh wrote:

> Guido wrote:
> 
> > I still think I'd like sys.path to be a real list of real strings (not
> > necessarily representing files or directories though), with a
> > configurable set of handlers that are given a chance to deal with each
> > string in turn.  (Or was this already rejected?
> 
> I've rejected it.  Does that count?  ;-)

Let's rephrase that as "-1" and I'll gladly add wy own -1 ;-)

Just


From just@letterror.com  Thu Dec  5 12:36:05 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 13:36:05 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-23473B37084E11D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> Why don't you care about the backwards incompatibilities?

Because it's addressed by using a str subclass.

Just


From mal@lemburg.com  Thu Dec  5 12:39:10 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 13:39:10 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]>	<3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff>	<3DEF3558.8010006@lemburg.com>	<m3bs401yyy.fsf@mira.informatik.hu-berlin.de>	<3DEF3B43.4080108@lemburg.com>	<m33cpc1y8q.fsf@mira.informatik.hu-berlin.de>	<3DEF3FD9.7050700@lemburg.com> <m3lm34zmee.fsf@mira.informatik.hu-berlin.de>
Message-ID: <3DEF48EE.8020108@lemburg.com>


Martin v. L=F6wis wrote:
> "M.-A. Lemburg" <mal@lemburg.com> writes:
>=20
>=20
>>Well, that's what I tried to solve in the sketch I posted earlier
>>(slightly modified to meet Fredrik's requirements):
>>
>>1. User programs register import hooks based on suffixes which are
>>    used to match the entries in sys.path, e.g. ".zip" for
>>    ZIP importers (caching could help in improving the mapping
>>    performance; this is where the auxilliary dictionary comes into
>>    play).
>=20
>=20
> I don't like this idea. How could I then make a HTTP importer, where
> no suffix is available?

I first had "based on RE matching" in there, but Fredrik pointed
out that this would cause bootstrapping problems. The suffix
approach is easily implementable in plain C. Its open for discussion,
of course. If you have a better idea, please speak up. We could
also use a combined prefix/suffix approach so that URLs can
be recognized by prefix, while ZIP files are by suffix.

To answer your question: web-imports could use a special suffix
to signal their web-import nature (I'm sure you'll want to apply
some more security checks on such modules and or packages), e.g.
".py.web".

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From guido@python.org  Thu Dec  5 12:42:10 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 05 Dec 2002 07:42:10 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Thu, 05 Dec 2002 13:36:05 +0100."
 <r01050400-1022-23473B37084E11D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-23473B37084E11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212051242.gB5CgAm12017@pcp02138704pcs.reston01.va.comcast.net>

> Guido van Rossum wrote:
> 
> > Why don't you care about the backwards incompatibilities?
> 
> Because it's addressed by using a str subclass.

Which strikes *me* as an ugly hack. :-(

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Thu Dec  5 12:45:30 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 05 Dec 2002 13:45:30 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-F5837BD6084D11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <3DEF4A6A.4010905@lemburg.com>

Just van Rossum wrote:
> Martin v. L=F6wis wrote:
>=20
>=20
>>"M.-A. Lemburg" <mal@lemburg.com> writes:
>>
>>
>>>But why store the import objects in sys.path and not in
>>>some auxilliary dictionary instead ?
>>
>>What would be the advantage of doing so?
>>
>>And, if this is done: How can I add my own custom import mechanism to
>>sys.path? The machinery to replace strings in sys.path is only
>>available for zip files, all other hooks must add their objects to
>>sys.path directly.
>=20
>=20
> +1
>=20
> With MAL's idea you'd need to write
>=20
>     p =3D ...
>     sys.importers[p] =3D myimporter(p)
>     sys.path.append(p)

Hmm, I was thinking of:

sys.register_importer(myimporter)
sys.path.append(p)

myimporter would then either be asked for a suitable suffix
or prefix to use in matching or the importer would simply
call myimporter.match() during the sys.path scan.

> instead of
>=20
>     sys.path.append(myimporter(p))

Sounds like the classical "you need one more line than me"
argument ;-)

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Thu Dec  5 12:46:04 2002
From: martin@v.loewis.de (Martin v. Löwis)
Date: Thu, 5 Dec 2002 13:46:04 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net>              <177e01c29c55$80b89990$0900a8c0@spiff>  <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <000a01c29c5c$46b61ea0$3c1ee8d9@mira>

> Why don't you care about the backwards incompatibilities?

There is no strict backwards incompatibility: Programs that work now
continue to work, as they will find only strings on sys.path. You only
get problems if you actually add a zipfile to sys.path.

It is unclear to me what the typical breakage would be, and whether it
could be solved in most cases by having the import hooks a string
subclass.

I can imagine a number of potential problems. Code may choke if it
encounters a Unicode string on sys.path (which is possible already), it
may choke if it encounters a non-ASCII byte string on sys.path, it may
choke if it encounters a string that is not a directory name (in which
case you simply can't add zipfile imports without breaking backwards
compatibility).

Regards,
Martin




From sjoerd@acm.org  Thu Dec  5 12:55:25 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Thu, 05 Dec 2002 13:55:25 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <m3lm34zmee.fsf@mira.informatik.hu-berlin.de>
References: <r01050400-1022-B4F3FCEE083611D7AD06003065D5E7E4@[10.0.0.23]> <3DEF2FC8.4060700@lemburg.com> <165401c29c4a$c9b91c60$0900a8c0@spiff> <3DEF3558.8010006@lemburg.com> <m3bs401yyy.fsf@mira.informatik.hu-berlin.de> <3DEF3B43.4080108@lemburg.com> <m33cpc1y8q.fsf@mira.informatik.hu-berlin.de> <3DEF3FD9.7050700@lemburg.com>
 <m3lm34zmee.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021205125525.44BFC74B08@indus.ins.cwi.nl>

On Thu, Dec 5 2002 Martin v. =?iso-8859-15?q?L=F6wis?= wrote:

> "M.-A. Lemburg" <mal@lemburg.com> writes:
> 
> > Well, that's what I tried to solve in the sketch I posted earlier
> > (slightly modified to meet Fredrik's requirements):
> > 
> > 1. User programs register import hooks based on suffixes which are
> >     used to match the entries in sys.path, e.g. ".zip" for
> >     ZIP importers (caching could help in improving the mapping
> >     performance; this is where the auxilliary dictionary comes into
> >     play).
> 
> I don't like this idea. How could I then make a HTTP importer, where
> no suffix is available?

Instead of a regexp, I would suggest using a function that returns
true if the hook can deal with the path component.  This function
could then use a regexp to determine whether it can.  But this is
about the same complexity as just calling the hook and let it return
quickly if it can't deal with the path component.

So this step would just become:
1. User programs register import hooks.

-- Sjoerd Mullender <sjoerd@acm.org>


From just@letterror.com  Thu Dec  5 13:04:13 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 14:04:13 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <200212051242.gB5CgAm12017@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-10C950ED085211D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> > Guido van Rossum wrote:
> > 
> > > Why don't you care about the backwards incompatibilities?
> > 
> > Because it's addressed by using a str subclass.
> 
> Which strikes *me* as an ugly hack. :-(

True, but you can't have your cake and eat it, too: either we implement an
elaborate scheme of registering handlers for sys.path items, or we keep it
simple and do our best to avoid breakage.

Jim Ahlstrom & Paul Moore's patch implement an elaborate scheme which isn't
exposed to Python, and it adds a fair amount of spaghetti code to an already
messy part of Python. I'm still a -1 on it.

Allowing arbitrary objects on sys.path is simple to explain, document and
implement, solves the zip import problem and offers a clean high level import
hook scheme. How much better can it get?

Just


From just@letterror.com  Thu Dec  5 13:07:33 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 14:07:33 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <20021205125525.44BFC74B08@indus.ins.cwi.nl>
Message-ID: <r01050400-1022-86DACA7D085211D7AD06003065D5E7E4@[10.0.0.23]>

Sjoerd Mullender wrote:

> Instead of a regexp, I would suggest using a function that returns
> true if the hook can deal with the path component.  This function
> could then use a regexp to determine whether it can.  But this is
> about the same complexity as just calling the hook and let it return
> quickly if it can't deal with the path component.
> 
> So this step would just become:
> 1. User programs register import hooks.

That's a great idea, but I think we should tackle this after 2.3 final is
released. It's orthogonal to the objects-on-sys.path-or-not dilemma, so can
easily be added later. The does-this-item-ends-with-.zip hack is good enough for
now.

Just


From fredrik@pythonware.com  Thu Dec  5 13:20:46 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 14:20:46 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net>              <177e01c29c55$80b89990$0900a8c0@spiff>  <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <194d01c29c61$20193980$0900a8c0@spiff>

Guido wrote:

> > > I still think I'd like sys.path to be a real list of real strings =
(not
> > > necessarily representing files or directories though), with a
> > > configurable set of handlers that are given a chance to deal with =
each
> > > string in turn.  (Or was this already rejected?
> >=20
> > I've rejected it.  Does that count?  ;-)
> >=20
> > </F>
>=20
> Why don't you care about the backwards incompatibilities?

hey, I always care about backwards (in)compatibility (and you should
know that by now ;-).  But I've also written tons of code that uses
sys.path in various ways, and probably more different code archivers
than anyone else (Gordon keeps working on the same installer all the
time; I usually invent a new one for each new project), and I'm not
convinced that there are any real incompatibilities...

Let's list a couple of use cases (covering roughly 100.00% of the
code I've studied over the years):

1) code that doesn't use sys.path directly

this is the vast majority of all code.  no breakage at all, even if =
Python
itself would add non-stringy things.

2) code that appends/inserts/removes things from sys.path

quite common.  won't break if PYTHONPATH isn't changed.  very unlikely
to break if ZIP files are added to PYTHONPATH.  unlikely to break if =
string-
compatible objects are added to sys.path.  and in most normal use cases
(insert/append directories), unlikely to break if non-string objects are
added to sys.path.

3) code that uses sys.path to search for things in the file system, but
doesn't expect all items to exist and/or be valid directories.

relatively uncommon.  won't break if PYTHONPATH isn't changed.  unlikely
to break if string-compatible objects are added to sys.path.  likely to =
break
if non-string objects are added to sys.path.

4) code that assumes that everything on sys.path points to a directory

not that common.  won't break if PYTHONPATH isn't changed.  might break
if ZIP files are added to the path (whether they're real strings, string =
sub-
types, or non-string objects).  likely to break if non-string objects =
are added
to sys.path.

5) code that assumes that everything on sys.path points to an existing =
directory

already broken (but I've fixed such bugs more than once, also in my own =
code...).
won't break if PYTHONPATH isn't changed.  will break if ZIP files are =
added to the
path (no matter how they're represented; see above).

:::

summing up, I don't think *any* code will break after an upgrade, as =
long
as PYTHONPATH isn't modified.

some code may break if ZIP-files are added to PYTHONPATH or sys.path
(no matter how they're represented), and some more code may break if
non-string objects are added to sys.path.

But as long as Python itself won't add non-stringy objects by magic, =
that's
not really a problem.

</F>



From just@letterror.com  Thu Dec  5 13:22:44 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 14:22:44 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-86DACA7D085211D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <r01050400-1022-A8B0C98A085411D7AD06003065D5E7E4@[10.0.0.23]>

--1439855-2295726080-3248086973=:9001
Content-Type: text/plain; Charset=US-ASCII
Content-Transfer-Encoding: 7bit

I've attached a newer version of the patch. It fixes two important issues:

  - fix the uncompressed .zip bug that /F pointed out

  - Avoid infinite recursion (and crash) if a *.zip file is on sys.path
    before the item that finds the zipimport and zlib modules. Thanks to
    Paul Moore for reporting this.

(Right now the zipimport module must initially be available on sys.path for the
first zip import to work; so you can only replace sys.path with
["myarchive.zip"] *after* you've done "import zipimporter". This could be fixed
by making zipimporter a builtin module.)

Just
--1439855-2295726080-3248086973=:9001
Content-Type: application/octet-stream; Name="zipper2.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; Filename="zipper2.tgz"

H4sIAAAAAAAAA+w8a3Pbxq79Kv2KjTK1JZt2JPmRxDrJGcWWG986lsd2etKmGQ5N
rSyeSCRDUnHcxv/9AtgnH5KVe/qYufdq2shcLrBYAIsFsFj9FsQxT7pPvvsTP+32
bvvp3h58t9tP98V3u7srvsXnO+jwdGd/r727+/S7dqezs7v/Hdv7M4lSn3maeQlj
3/0b/ljWD7qNx38FQX/t5zcpf/gOZnGUZNv+Hz5Gu9Nu7+/uLpD/bqe7v1+Q/84e
qAtr/+GUVHz+j8v/cRD60/mIs8b5XTaJwu1Jo27a0iyZ+9mMz655kn8TpSM+TvNt
My9JJ9403+hHsziYcmys1x8DUBBydv6zO3h3xdpWwyG1dKyWIbV0dcur/hG1bHUA
E4gjC3wWhBlLuZf4EzdKRjx5/4G9YL9LbI5C4sgB73s23MT7zN3fpsE1gLR7gPPJ
BtPLgCcsuv439zNGowdZEIXMC0csncf4nm08qdezu5jDaybYlAP+JYhP5N89pLbU
4fd67fxuSEO4rwf9I/PINnA+wWfeYwxIir1sEnozzqIxyyYckTDZAYmwwLLIBxD8
AFjmXU8Jxo/CjIdZykaBnxUg4gQm96UnIBrj+XR6t/1p7k2DccBH27GXACB8+R+9
G76NRGw3EMN9Twvg/O4KmCDxWZN2sblHPB1Hya2XjICR/hShNaSi4oZnLtDe9Cew
EtXkW71yx2nkjVyYvzuLRvMpb1rjsY2UT8cOEzhwJkiuU6/Bh+VY5MKckjtArxXL
Jvtwwv2PzShuaSCaiWp22Fpxkq16mdC6JWk35LfNHJs2/GnqMFvcN7nnj7ejtFUH
BSnNr2cLbzSfze6gRY4uZv7xdhqkmVgGDcnKhsMaMHH8EgKHv9oow1owZs1H53f9
5MY9h8XLr+bxlPfD0Y/87hbWU9oUpCFBADx89HU4PLCm1hD8rTExKjDn/O4yS4Lw
hjgDz0Sj6iWf5HerBc0Jz+ZJyM7enp4iPThHoDwv2Bawa+tlFrvedIpagsxrg/yI
eoJ49IIwtHBREY6tl2qFvJC4VTvwodQmmGKa7+uKrqbhdkvw/942PTkxo5Go0sgV
5Pz3SXDbJbpd92FR5viqRaq5qnsIAm3pbnWQ0vM79+Ts8GJw3MxhapFOu+9y7yQO
S8ZScmVBC4EqE6KJ7qfij8JgiHIBTpyRRTBqAZ+mnIYq0g5wrZyitG3N+BwFo5xq
jLhQ3ZJ2kPRx9keDxZzJvZMjl9oNx8RzdO3i7kTrZpxwTt1aNpULrBVYxJGyr6YL
7IVFTa62UOXVC6A5s0V22ZjohUosNRd0NqewFn1CZ8WOV1JXxF1lYozw5bKXxoOt
rbFiO3D5LAq5UDdFMMP2szm6RG5/NMohcxgNq5TMgJR1zNgarWXWCIIztuKpl0Lv
xDzO7954cYyTfu2lsOKb1mrU/VsVGkwULrRxSBTs24LJLIwy2MLn4PoI30HhUczp
6UUgW1ZQMtrI/0wl0w4AbPmOte3D3zCsMfXoBwbfqoIW9UtV0IigrIaaIlIn6Wn8
wLOTjM8qxShtoQVndKpCxWEuNAXp0Bi/h339ytS7y+A3br8Cld9R6jJIEveSZ9KK
wvMXn+YF7VEizD9uSeyzN53zlM0gfGLXnGWIOEW/c8rDm2zCdhqkbDdRFrERaQep
LziFrBmQ580C9g+2A1+bm/bgh1Pw65sEnBGTBM2KR5ZIA73gMuSKUkykEZ3fIJzz
XAda7sCGIo8qeEM8eSrIWo0ptQaf8hk53Oiwr8Chp4JDORbhEqwJTS06vcqrKG50
WlMclmmGIIpHlvG5Trj30bIgNIQxgI/EFIe+P08SPmqS5lZPWyw+MXGI8rxwPSNS
ldkYJ9HMjlVwkji5A7Ihcu+y3XGlxQCftyBvOASmoyOIs2wTMqPWVPhJ9drvjdzG
wBZsaQ57M7h67f7Uv+hf/HAJhL97965x7yB8blWzBdZqMTyy0KnV8OserWfKUfP4
tBQvvaFoujwdbLWmYzy/K3f46r8GhxDFRuNxyrNobBs/RzG4hS4hzUR4ivh5CBS9
CdFTgmrv8kFQ6XA4CpQmTjM9vzsCsV5eXQCXmjk3CAxaHTkGWu5jWFgRbjfswOxo
cDy4uBgcuf2jo4vB5WUTv1usvVr0KTiZC7Jh7zq5apbQrgk8IpwDlW/DhBomI1aI
eFKwDHlutByEodWPkXfsBmChsBttmc0RF/F/lLQqvEJHwshHAskhixPcpUrN4PJ6
WZaUX6SLXmAqBkL68ouExxXdvdQNyc2pfJXyT3Me+hXY4OVMuCXldxMvnVQQBjOv
mIecg7U9hjwJfNgB+jA/p8CIaCEnKt4Ajdfz8ZirEdyr8+PT/g+XYJiO+29Prxhu
BLrxVf9ycPXz+cBhAno89W5Sgiyqtx5gVCXJLPE+wyKv4JmPm12FYAJ/slBqt2DK
MVYT67P8PsiqRIetIf+SlciXBtVMQTZUdCRTZXekhkoVraTs2qvigc5P5Vt56ieI
adGryhEQl8WXJga5cQLWrhixay3CB6kMZAuksvXFEtV6o5doIcNjusBDXm2P+NQx
lGEwpvYEaAEcwl3wKMOIWUUrv4BhLchKZsYwE0gOsXgcRRnsmvAvGDpYK7ALJE3s
4rD17XXlL9LrkqcoU6k9GXyQwwgo/FmMALCtbcd3DdttFfnTpf39AsDhgxBRAWIo
IYpE5nIuFOrLlC/s6/xLExuBScQZ6c8vdS9FBGjljIMPyCJCYeKzIBffY0bgHgVW
h71SJRvUxsu2XuJOWsd3P0BDyDzK2kIj/GX5QA67ngfTETSifjpMmUny3e6Yzry2
EJMKv8ClSxETupLoKwU83aah+swEA0EKSLMJ6NYW+ZcO454/YbgPoad5OwnwKWU8
yCZgFdA/ZsAiD/GkHEgWYAeEGD9NlXYW7gF7TDNyo7GrMmubrPGoAf9SexCqdoWA
od1KeJpqBKoB9fwjiK7H2uQEzEP1ho809MjLPBe3UEdC03YKU7F7Uy8NMg7Aba8C
WQogrIRDANJiAAj2Ey4sHg9RU3F+QRp/vHHUE0BfR9EUXKpkTuqOqXopQhKOzKKD
5BONAs2w9XmM+g3+VkhUiyXnsOJhhoJukayuYBiSO8k6VYcERjNIrgw3ECn7FG0W
6GBqRH3nu1Y4BQsz/2ieSDHR4YcJ3vJ1QJrMwxBVOAhZFGfBDFhOIQAIITdCAaW/
eAQEOosynJaH6aIIzGYw4yiIWcx8DNVowJRipQMkxWMbaIBIoPiC0hWIh1BAQzSF
dY5PIXEHezv6Lx97pFkAPgh6+ny0Xa86pahXH1PUfy8cwRRTDibZcHxyOmAb4xj+
nEYwBbNC/ATALJW3VDmnpNAFmpAdCseEA8WJ7oArVgLKN/igOgcOAw0VYScMClzK
ZA4ENNERGt2T2wsu6/dv+u/O+1evTwdnsND3Pqh3OMqidxuxo7YmMrRZAuMZU/mS
GbhlqYYhOErjaXRrpxvQkpJdpSRZgFYxYjixhp3PMok13G/iO7knmqQq7A4xCGUc
xYYw2JCS64baNPG9nWleEAEPVfSrfYEDJgJhRE2GH6W3gDoQGP8IQzlsq9t1WLfV
Qw/hEhqF4eHhCFcz6TT6EyrhR2es7gVI9xSmfgxdj6ELIKJcRftLe7+9t/tqry1I
B5SvvNEBGwhsh6iS3pQdBWAIgpvQA6IE+trYn0Yp4emtGPTjBNW8MUvoPTBlynpR
KucItj/3jN82rbTWkoSWxap94lS9ltN7wrmMMz1KaF4qU25xgWNYdkcckOmYF6yJ
OtWSmktOFyo3rhZxUiy8i15P5oU1cYW12G6JA1w9LIlSdBIsn65COPFnKmXbbXdI
tiaXoyVsixbB2WtrpAVEwsrttOVpWk2ZoxJRlxMQcZEqsskr9USjtVpPMIMrcURb
ypV6a2O6Um9tQVcj2bKyALC7Dxw1GDbrlBt+AM2K3XpLxbjbVWK0NoyV1cuQnLPP
qGU2P8wrXJW12DqoKDrcGkw43ohpI97cxNWF20QLNlNfUbARExxqcn80QsOHG3wI
PjG7vsukeSpM+EV+f4MO6EyJg0CMmHLnMPTqhQooiBgYC00WOQLkAwD9GJL0WPoR
jBgNaWWPhSWlPDH64jPvIzgkaDrRiYCQENadvbphb/KjJOEyoLXkltvOhcC+3Qrs
tne0FdA0akNwSuQsMQG2hmyy7r5NyE4bmv4ghUWKBG8kW3RKLKekIMtpr1YylOSC
U2dyRYR1xvTo+qN1OrvPklBt70CztN5g0IQX5FjayrbU6y3WaSlpitMvHKslPYqQ
AjYr8BLj50NsgZxCbDLv3vTWuwMHcu77nMt0SW1DwKz/2l7XJ84l+Fxmng7cSKHA
ScUQADEEwl+lRVatmPlVCHEeUN7ciOWJBY67Qc7M+pN1GkYsNhidjhxgRarDBkxZ
w1YfM1URIIYin1Bue4sZIaeI73O1GAghhqiVMwC69EAeUVLvra2eIDPPQENIhx7v
iWpB9gWHtTbK1UWJ8C1Clx+WohQozSd/7kZ+iDxRko4OefA561E+crOPg6xjqgJy
9xUG+z/h6U+zMRwOG446mir/0dI8kkRdVhPlMOu47h9MmABxcsTRK+sJwirPV4hp
WZm0NKAPkCc8ZROWmIiEhmEL4xIZC2MKpXTS1qqXKNRnbOWzvFJ2B3GqE62KmEr4
ZaRj6FPbPqzgpvsTT66h8Xjq3YjTrMu71P1XAuHyZQZGKWk2HjPbk/VgC/p+OpKJ
F3LFv09/DcVJL41mBxPKVcXauzpNsHjAVSjXqDryEnOpPDVHfoy4Eok7nod+E+tL
CnGnUHHhugknHdX6DbWiPjVbueNxrHpEMyirHysXgkSJfj700rGRAFngqptKAHOi
jbl6iRJhAZ+ZT2NRRQpOG888UQPzATeIQaRfwBs3mopPVvyMj5au6kehrwXeJd4t
jYNxK9oNMTd80nTq4PZ6Ps4H8hQ8JxANoaeSuuCSj2Q56YNR5qNxbIeXx1Ey87JC
bCk22kKASZ4LhZi0UX6fNnIaWRF3AdnK82IbLTqDdN+ImiTjS2/S3kikUf+HI2BA
A8FTIUTPR8I0CAYA4rhFBIb5ULNMLjAUWbfEaSKvDnu9EDYwx/0xfuMcHHQG7LQK
DVgyEojo61dbgrCJabBVUgALJk+4EFFDxtTI92OsxgLiFswdCdKRGLp7YnwAeK9J
Ii/ol3Xhf6TerdjmAko5IK/AkZWei4JQBlLpOi16WUGAPpv4sx+OqOyBWGcYQKaj
SDrSabAtrUUxgSVNB4kmKKAZNdlKzxLREoNCjlgA4q3O+7LbADw5skMqL/JI13Iv
lZaVvVggMZNdJhJ7rCE74miU4/jsBVMsrbbKWtS+Bvw1JkPGIkXbrc9luM0Vxbwc
Oi0naUsPYS0dAw48KLLgsT5pChbAmLGtzp7cdOSeYzq3qjYpBWntSILxlaZ5Hso6
f9ePRrxZKLhm5qhKPFv1yjhE3vQiBtu22mqpC1sUaVTrrwJz2YnUVXWgHCfmlmfe
TeAX90JsE2V7zZYuBER0/3jBnn+b2lzDssaY0Szt6nWMa+V9+wNbg6BtPCZ/v2kR
qJq/fhV7GnXv5Ls3TfeXLxl71qqG6i6D6uwvgNpZBtXd1VCLt6k8a2hjbrzCIiDC
IssHwDB9v91tt1PpYmrvupQTRX0oRcNCWYyVot1pkz1zhDZsAVN6ytCM+GLfRJbF
HUInWfGF/U19pFwN1Nirnm+u1kvMFl1HecdF1z6J2Sr75jGiS9a3aMjlnJANYoFU
rsNx8MWdBiF3Rb4ktQop02ie+Nw6Kv4E3kxcubhkV7m8dAo0v8BMJ6vCMvjCR654
o5dTJg6hISr8NYH4tiwK09+Gr96NmrFep/nOCzebT5VTtIFbxeytFSETyRTZbXyi
0DNc11HZRjMm/0h0DFVHFUDXZPJYRq/3MtjCo2+NbyPuFcZrr+dq8ggZ+6S26qYu
ZtflubmZUI23tIdidGO+88KpVB+ps7LTN5hxS7nyhtwpK0VByCWVtbRvdSlrI+Ee
ijloa71U8qaWVlhxgCe/MgjjeVZJAthECVswEUWNWmm9fvvlpcp7S3m+y7hFsh8w
Y9gCsZzeVJUE5QKviu1XiOo1OnOt5qGbFmrpaoiHypULpddWTkBlKA4KLMwVXxtV
XTOTXLNmuWamaYHlJrwmZ7yGSYcKSqVHlgtO7Qsx+bKHBzmsfcHlLnQK/q4/oSw2
KaKP1VOiOuCAEiC0HArr2SiRYYzyk7TFkZgOJSr5OCxgLjh8SzGXdmFhJ6mEcdxs
DC4uhhcynZLfAOVue44dmzpbpQgd8bE3n2YHK/louXAZXTVM/1MRdkOec43se2Gm
JNr2mJXP+ZBXUZdV2trVHHzhProYIvcy+GLxS6xSe9plr2PlpJWYIPASS3MSrl0Q
XXdtklZlienVKbKqa2vMLhRXB8eYCXNdqu9xqQLJlLRQoGbdI4U+WH9CKqAyIjgQ
bEzEcqoooIphq27gA2XiZTeRWMYTV/vKgz6dwh6brMtevigVEKzqtufCPX3XJl9D
UEhU3tvnDPKGj+GmpIsA5eRk0weTY8+/QCfig5XXVoxbGOfZt3NKV09lODIcomed
v5/HrDsk5m4YsdOMuTBBS6WY5IeTuDF+oiwiSJ/QiARNZapaVLY1lN7oe4WsmHZ+
P/wgjQAS05Iut1oPul1RjUM+ktmeUthtXx0ox7eS6gXJVsymMjqJwCC/KrlaypbK
pS6+BIuaKktaE0WduDZzKQqTnLDvnXfys6ZOvfytsMI19fK5w/03pbpdQz7MGTTq
QNBpzIUZ8J+sYfIdB6zx9qz/U//ktP/qdFDM3NqXNGZx6XZG7nZE1fWIwqWBWbzs
toC0QnRXgC5bovj02wX5ce0ZKXZRmS8GmHfN8qVudbgilMYAuUNZ5UZMVpbygsu6
bjpKxp8JwL9E3XktV2vaRo6YUtf8y454qSpn763dhaQlVG23aQwZ3VQx/JZPxDqx
6EQq+/znq9fDM7d/fuL+NLi4PBmeoeI8sMrJNaEuhQTVGFTCcGSxDcjdm2BLLZnm
d34AvSpR0tgK6xr17u/+1Yz/PR/r91/wexvsrj/5g8dY/vsv3f09+/dfdvbw91+e
Pt35/99/+Ss+J3jIegDmOJvHYLLqL/7zT/3i8JLCnAP2xP+cJlGUPYnpx2XU1yhI
sydp4j9Rwzqf62Bok4B/xsqfBL6oSryz3el26qNgPGZbPttK6NnQurGxoR9q3afs
LPrMuqBQrLNzsAv/7bMtVLCaQLO1tWV677Ej7uvO2P+p6IxI7Q8N0n2+7+y0O4wa
EI9o6DL4e0tXc1M9d7rtwUIKR002+JLxEKfRXHcnUZbC/+sOe68ftv31Dy3WWgUc
b/qADyfA5cPq4F/wGArTLikhMI9iF1WINh9EpO054bF+MUpRYhHzmHlJ4t3JrGe6
ApnUnzDTX3nipJqK3yh6IsYdTbcnf7G+lsZfoLjd7c4zS2/xkVlEo06VUNW6++yY
X0ul7Bzsdg52dqUGEz5UvDLUt2lyZ9/pdoUeM1Y7//EH9+jkYnB4Nbz42cGWQ/fV
25PTq5Mzejr/2T2+GP4ygKdH9HQ4PBpcDC6Hby8OB3iM98bzWRRO6bSRMfDjSAfk
rx0hS+mKFnhJSDuOvaOWzH82tlMcHLqcvDl3Xw+HPy4npEqRtv2/U422/YVK1O10
cloEz0yTXFKibb/WfW6MYLcNqnCw29YqhNhKOgRAZQ16tkSDnrX3nWedHWMLRcO+
EmydzneP5W8+kV5U/46TTIOKvKLDRE2FyoFSzQY4fZtLgbslaNNNYdDDl5Rhw/6p
E4XIQoBaqBKF6jXm8NxMjbdhU7gUf7dygE0zgNW+YaMvku+Ok+g3Hkrk4kHiljkT
JYJTvIxv7h1hUKKutmW6Jk7+xtm/Tq5eM/pdjzeDs6vBUbXknz9znrctyT9vd5zn
O3tK8mV2wk5TymTLGjN14qtYivUhUx4S05EdkgOxO45xZ/kdeWWQTGA2U7yFrv4S
V5hVUg/6lqWR8JTeyLw4/o2PCVVN5URlyBbErunhiBzCQuUP3Dq8R1nmMgwCfYE+
neS5gOemfGkw5nubdJxAntoTzOVA4Pke/6k+47OxmrHi/8kA8koNCU5wQqZbSDwq
jZGbl1X7ViZE1XoI2dwLpi1cUPCuYk0t1jJrTeHVcyFEHLakYZvAA9A8uj7qsBDB
cSXVpHKiZpp8JqtSr/Eo1tyrWj3PO23nebdtrZ7drvN8T6+e2qJZj0cu3hrNgpB+
EQx/Awz+15vlfe8BYGUALODc3isQVN3r2vzv9o61p40j+Jl/caZKwbGh3OHDdlBa
VSqpLDUP1UFtKlWWMSaxANvyGadU4r93Xrs7ew/HJJSm0s6HKNxjb2d2dmZ2Xo7/
dDsG8wWKt7zN61UINGIc9xvgNG7mcL775t3r/uB1P6krwmbD1fh8YKmqL2py80Lz
dU4TYULDBzCUeIFTGWbZeLHctQtt3dkiEnU5mmGZYj0ayvyNKtJK17h12Oy2uta+
IveuN1kMCOMNjjU6Cc6TUQ74GqoGvdusdHclyt9emPf5+p2afXWaBvFeG0R5p2V5
70sn6rZXRSj/wWZfSvYUyH50aMl+Z1absXJzNQhNMrOldm2LrE0JLi/qOVOqT4FI
D7KcvFodWK1ux67W5+P3uev0BUg/FGuUr/xR2uy2lVCNwWxtorvJsfYFK8XfJtPB
C5jur+P3YIaPF+NzcdxL0BjkN5bnXzRFmMmUpBy05hN5i9QG2g1z1pWfRnN7O4ck
fNBhJ2KsFMcOKI5OqnGEk1R8kFj3Q7H4ivSX6XjA28plasxv1U7Xt1Y09V+APLbL
FuvRCStrL/vHquHBh9nsUmQ9NsMFfX/a7736eXD6qodnNkvGN7en08nIZV+t6pac
Min3xMmUYs3uwo99Mx68VyEGukCoblsTKmkDoVLFDMy7It2c7jDVL3D8ym7ORleg
S/Do0pCZ1+yC0sxP/hqOljJ9Wvw8NXI93VQG/AoUsG5UJYR1YlaPUvPtM01/Y8xu
GbOtPHqYHxJzKbat8DGDurQW/2n7SKmxu1Wwd/WjfvdEeb7KSsGP8f+eR+bZLWut
mIP93bG96e02RQsrQOyI5s5dAVt4zSAoN4ktzGNejK0R8UOWl3McgcxgONlUZsnT
bBf5+XQrJVxMYLsRuRAxmCT90xcver/3e3+cYMhbRJLZLsqkxp1TL7c4QQwmKAu9
HXEUN+P4QGn+ksF8RPAsuaIu0sPJNKPINR/rhSQNhcj3UQuTCdBWhD/3YmqssjPf
0RcTvjjBi/QyGuPm5iHf/Nt7o8UX93csK1FXaWxJiZ0fTFsAPPGjCUjsZLavDuNS
oIwCgSoyhWdI2hlP6SSDXafM2Y2+83aMDw4Xk6tb4LT51XDERZ8UQadM9OGUD0Ei
QfhNbD4xi4YSXKYO4mN4f3SzQN/OftS7EE8Oyhw7HfsuxrqwrGUJn7i6whPRbTSb
RtlthhGaD/sWPTvpvNoRE8psjO0nGTCB/dDedLbcc7FdUnpaFqlh/T1Ph0JFPnM6
NCK+L23BbG2XrEwzskNKrO9T47md6t6UW2Yl1wTjbYBUY+Xe85HSjd/9LH9L5zUT
1U3sKgW/Yz4vLFovIGse9OetvrDpeuQVkacYvA2wjaWJq7Kp2If0bHjgh2ILUjT3
5QitnSaYdNvIy15/gGd2O2M+wYSMFJUsgSkykrY0W/ygREdBCXjfENONzZ7r4Qjr
h7MPJBm/4/MfSOH+fDw+v5k/44ZMZgOz+KCiHerHBnwB+qxKjHfR3IvdsfIpij9s
aZPBiXYWfcRiYaxewcwNFDtnM+r0s1zconRZYoLT7NK8SVXOcP+acipAC1LCYFbf
5wfIl75lRUmP5tebvkHBh40S2Tw8eTvovT15ae1DIA++pi3I/CPHyhB8ORyJMc5f
l21bzHhembJXlE9W/1XZEYCM++O5+G+d1+PdwMQPTICgWv0ZQ0IPmT/+ODww4e/+
uJBGBpUMGhlW12rkf391ZU+WmLTm1PaZ6/8JBqhVkO+rZIPNjnWsNx+NZcoFRHKI
AqKj7LwYL8VJV9l5HgV0MO0xqbDmnAtWKaLR8tBotQCN9MChgcYinVMoN0iiDfh5
NBgVYBYebpnM9jOQokgzCCY7D2aX7I3R/l05HgG1qbsDGJmUF32NiTHjzAzxkKRy
TlpNKDyMlNMpRbM+1WZ9nCKdjjak031o8EjIVnNFfJgCai3t40kOO804aalDjHcQ
NR6ynI9m88n6DppK5xMwJzBsK1V5IbAO7Wactq00L5aBRFEx+sl73FSVe9ExNEuW
9CszVUGwtdh4L1zn40xro1dbmJTsBVt5mtR9yATEaGomzFMW/TGZucXwjqJJIyoJ
6uaJUhL9c8SR+E2hIcJx7uqxifKAqaeigTqmwGc/amGBTaRrz60DgslUVabq0GZp
ilHwySJbcstCbkWIinu8k+GR0dXv71IzxWgKdmJE/hKRMabugzCskv1pcgAMl2j3
4FELLh2lanu4H224lu0oRQ8R4UryzSD5TM7YuYD7ereWjq0xcHGk7qkuQoNcBW5E
7xhR8CdV58XbIYouLcFNQpaIri0f2STCEEV2+j+R3XU5nX2EY8pHNLVM1QUXeDKH
UH3Ik/P6drlbMu62QYJ1Oy7tprgeJK8k47vmEZ8jkXitKgqHA5aGI/mW2UDK58sp
bKBqmwmYn2vYpHpaLgBvQ6X3nUQ5rbox0KrbcbQS44mkcI23HydRcwkFXuLQq04/
yG7OdACfJKkmG/r753Xj/7a6TXr9Wb//+TwfXTHOP2Qh4JzxHJnxJZsGRWaq14XU
cRdInaTKGsthVUAhyePgkhAea+plCwRsHzfhHxXjLP1JGK3y7c/CMMY12ddaDbqa
mgsThQEM9r4X9eICMeJl1S08zE2pYFJe50jO/L0FGKNz4FusQ5QGudSuK5N+uXAO
x9k0ua/b0IqAxfgC5DU2ucetb9tx8e/i2cwayVSUpU46sNStnIP1C+mT5AhkWUET
iTdoUQmvJZ6RyNdlDntXyYKCNqMmwtR65uxq8h6UFRKFPa5AFeu6FLeLLYanEyn6
PUQlO9+lMKT/G0b57kPe7xgZ53xJzUHhBVhN379/9/Wwyn+d9R0gQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE+D/DP6bqvacAoAAA
--1439855-2295726080-3248086973=:9001--


From guido@python.org  Thu Dec  5 13:24:04 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 05 Dec 2002 08:24:04 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Thu, 05 Dec 2002 13:46:04 +0100."
 <000a01c29c5c$46b61ea0$3c1ee8d9@mira>
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net>
 <000a01c29c5c$46b61ea0$3c1ee8d9@mira>
Message-ID: <200212051324.gB5DO4p12167@pcp02138704pcs.reston01.va.comcast.net>

> > Why don't you care about the backwards incompatibilities?
> 
> There is no strict backwards incompatibility: Programs that work now
> continue to work, as they will find only strings on sys.path. You only
> get problems if you actually add a zipfile to sys.path.

I've written code that searches the path for data files.  This has
been brought up as an example before; Zope 3 for example looks for
page templates and configuration files on the path.  I'd like such
code to continue to work even if the standard library is loaded from a
zip file.  Such code is typically robust if one of the sys.path
entries doesn't refer to a valid directory; but it may only catch
os.error, and not the TypeError resulting from attempts at string
concatenation.

Obviously if the data files get stored in the zip file too, the code
searching for it needs to be made zip file aware; but if zip files are
used to provide *other* libraries, that shouldn't break code that
looks for data files along sys.path.

> It is unclear to me what the typical breakage would be, and whether it
> could be solved in most cases by having the import hooks a string
> subclass.

A string subclass would work.  But it sounds like the requirement to
subclass str makes writing hooks more difficult compared to a
mechanism that places the hook in a separate hook registry.

> I can imagine a number of potential problems. Code may choke if it
> encounters a Unicode string on sys.path (which is possible already), it
> may choke if it encounters a non-ASCII byte string on sys.path, it may
> choke if it encounters a string that is not a directory name (in which
> case you simply can't add zipfile imports without breaking backwards
> compatibility).

The main breakage I am intersted in avoiding is brekage when you
concatenate an ASCII string with a sys.path item.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Thu Dec  5 13:27:29 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 05 Dec 2002 08:27:29 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Thu, 05 Dec 2002 14:04:13 +0100."
 <r01050400-1022-10C950ED085211D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-10C950ED085211D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212051327.gB5DRTC12201@pcp02138704pcs.reston01.va.comcast.net>

> > > > Why don't you care about the backwards incompatibilities?
> > > 
> > > Because it's addressed by using a str subclass.
> > 
> > Which strikes *me* as an ugly hack. :-(
> 
> True, but you can't have your cake and eat it, too: either we implement an
> elaborate scheme of registering handlers for sys.path items, or we keep it
> simple and do our best to avoid breakage.
> 
> Jim Ahlstrom & Paul Moore's patch implement an elaborate scheme which isn't
> exposed to Python, and it adds a fair amount of spaghetti code to an already
> messy part of Python. I'm still a -1 on it.

I know. :-)

> Allowing arbitrary objects on sys.path is simple to explain, document and
> implement, solves the zip import problem and offers a clean high level import
> hook scheme. How much better can it get?

I fear that the requirement that all handlers are str subclasses
raises the bar for writing a handler too much.  But maybe that's not a
problem.  (It's not like writing import hooks is currently a piece of
cake. :-)

So I hereby change my position to -0 on str subclass handlers, which
means you can go ahead.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Thu Dec  5 13:33:57 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 14:33:57 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net>              <000a01c29c5c$46b61ea0$3c1ee8d9@mira>  <200212051324.gB5DO4p12167@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <19a301c29c62$fa298bb0$0900a8c0@spiff>

Guido wrote:

> Obviously if the data files get stored in the zip file too, the code
> searching for it needs to be made zip file aware; but if zip files are
> used to provide *other* libraries, that shouldn't break code that
> looks for data files along sys.path.

as I might have mentioned before, it would be really nice if custom
importers supported optional "load_data" and "get_module_names"
hooks.

the former can be used by code that looks for data files on the path
(Zope, PIL, most likely hundreds of other tools/programs):

    for path in sys.path:
        if hasattr(path, "load_data"):
            try:
                data =3D path.load_data(name)
            except IOError:
                pass
            else:
                return data # got it
        if isinstance(path, types.StringTypes) and os.path.isdir(path):
            try:
                data =3D open(os.path.join(path, file)
            except IOError:
                pass
            else:
                return data
        # ...

the latter can be used by code that looks for plugins using a
name pattern (PIL's *ImagePlugins, for example):

    for path in sys.path:
        if hasattr(path, "get_module_names"):
            names =3D path.get_module_names()
        elif isinstance(path, types.StringTypes) and =
os.path.isdir(path):
            names =3D os.listdir(path)
        else:
            continue
        for name in names:
            if name.endswith("ImagePlugin"):
                __import__(name)

(stupid idea: why not change os.path.isdir to return False for non-
string arguments?  it's a predicate function, after all...)

</F>



From just@letterror.com  Thu Dec  5 13:35:00 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 14:35:00 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <200212051327.gB5DRTC12201@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-5D72C28C085611D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> I fear that the requirement that all handlers are str subclasses
> raises the bar for writing a handler too much.  But maybe that's not a
> problem. 

/F's compromise to the rescue: the zipimporter will be a str subclass, but *any*
object is allowed. This makes sure that *we* don't break code, and makes it easy
for others to write import hooks (really really easy if they don't mind 3rd
party breakage, slightly less easy if they do, and somewhat less yet if they
must write it in C).

> So I hereby change my position to -0 on str subclass handlers, which
> means you can go ahead.

Ok.

Just


From mwh@python.net  Thu Dec  5 13:43:38 2002
From: mwh@python.net (Michael Hudson)
Date: 05 Dec 2002 13:43:38 +0000
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: "Fredrik Lundh"'s message of "Thu, 5 Dec 2002 14:33:57 +0100"
References: <3DEDF004.25379.630FD638@localhost> <r01050400-1022-31ADF6EF07CD11D7ADDD003065D5E7E4@[10.0.0.23]> <15854.29224.574966.648231@montanaro.dyndns.org> <3DEE7B87.5090602@ActiveState.com> <037001c29be3$ba396400$6d94fea9@newmexico> <3DEE948E.3010802@ActiveState.com> <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net> <177e01c29c55$80b89990$0900a8c0@spiff> <200212051231.gB5CVAG11941@pcp02138704pcs.reston01.va.comcast.net> <000a01c29c5c$46b61ea0$3c1ee8d9@mira> <200212051324.gB5DO4p12167@pcp02138704pcs.reston01.va.comcast.net> <19a301c29c62$fa298bb0$0900a8c0@spiff>
Message-ID: <2m3cpcwph1.fsf@starship.python.net>

"Fredrik Lundh" <fredrik@pythonware.com> writes:

> Guido wrote:
> 
> > Obviously if the data files get stored in the zip file too, the code
> > searching for it needs to be made zip file aware; but if zip files are
> > used to provide *other* libraries, that shouldn't break code that
> > looks for data files along sys.path.
> 
> as I might have mentioned before, it would be really nice if custom
> importers supported optional "load_data" and "get_module_names"
> hooks.

I haven't seen that mentioned before, but it certainly would be
useful.

(pyrepl currently has some nasty code to allow tab completion of
import statements, bits of which I'd be most happy to see the back
of).

Cheers,
M.

-- 
  Hey, if I thought I was wrong, I'd change my mind.  :)
                                    -- Grant Edwards, comp.lang.python


From gmcm@hypernet.com  Thu Dec  5 12:45:50 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Thu, 5 Dec 2002 07:45:50 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <200212050911.gB59BIJ11246@pcp02138704pcs.reston01.va.comcast.net>
References: Your message of "Wed, 04 Dec 2002 15:49:34 PST." <3DEE948E.3010802@ActiveState.com>
Message-ID: <3DEF042E.22672.67469833@localhost>

On 5 Dec 2002 at 4:11, Guido van Rossum wrote:

> Note that I've seen people write
> 
>   sys.path = ...

That works with iu.py.

> I still think I'd like sys.path to be a real list of
> real strings (not necessarily representing files or
> directories though), with a configurable set of
> handlers that are given a chance to deal with each
> string in turn. 

Which is what iu.py does.

> (Or was this already rejected? ... )

Evidently, but I've never heard the rationale.

-- Gordon
http://www.mcmillan-inc.com/



From gmcm@hypernet.com  Thu Dec  5 13:46:36 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Thu, 5 Dec 2002 08:46:36 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <r01050400-1022-4654686407D811D7AD06003065D5E7E4@[10.0.0.23]>
References: <3DEE7B87.5090602@ActiveState.com>
Message-ID: <3DEF126C.22266.677E3DF5@localhost>

On 4 Dec 2002 at 23:32, Just van Rossum wrote:

> (I'd still like to see more examples of code
> breaking if a sys.path item isn't a string. Not that
> I don't believe it, but I'd like to get an
> impression of the severeness of the damage.) 

Greg's imputil put objects on sys.path. I know
of two imputil users (at the time - me & /F) who
got bit by it, and *still* wrote code assuming
sys.path consisted of strings.

It's Python folklore. You won't get users to 
change their minds.

-- Gordon
http://www.mcmillan-inc.com/



From gmcm@hypernet.com  Thu Dec  5 13:46:37 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Thu, 5 Dec 2002 08:46:37 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861996F@UKDCX001.uk.int.atosorigin.com>
Message-ID: <3DEF126D.31051.677E3E3B@localhost>

On 5 Dec 2002 at 9:07, Moore, Paul wrote:

> I read Gordon's comments as the latter, which
> implies your approach (A) below. 

Only the first sentence of (A). Having things that
look like strings but aren't on sys.path just means
that you have to convert PYTHONPATH, and trap
modifications to sys.path. Why bother? There's
an easier way.
 
[Just]
> > A) Stick with strings. Hooks can be implemented by
> > subclassing str. 
[...]
> > B) Allow arbitrary objects on sys.path. 
[...]
> > Regarding PYTHONPATH and
> > sys.path.append("/path/to/my/archive.zip"): for now
> > I'd suggest that the sys.path traversing code checks
> > for a .zip extension, and replace the item with a
> > zipimporter instance. This check can be very cheap.
> > Later we could add a general extension-checking
> > feature, where one could register an import hook for
> > a specific extension. This might be a case of YAGNI,
> > though...

Here's what iu.py does:
 
The first time an entry on sys.path requires
examination, the list of importers is traversed
until one accepts it. That importer is associated
with the name, so subsequent imports are only
one dict lookup away from the importer.

Search order is as expected (sys.path.insert(0, ...)
works). If your app only uses stuff from sys.path[0],
no time is wasted finding the right importers for
sys.path[1:].  

-- Gordon
http://www.mcmillan-inc.com/



From gmcm@hypernet.com  Thu Dec  5 13:46:37 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Thu, 5 Dec 2002 08:46:37 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEF2FC8.4060700@lemburg.com>
Message-ID: <3DEF126D.28673.677E3E81@localhost>

On 5 Dec 2002 at 11:51, M.-A. Lemburg wrote:

> Here's a sketch:
> 
> 1. User programs register import hooks based on REs
> which are
>     used to match the entries in sys.path, e.g.
>     ".*\.zip" for ZIP importers (caching could help in
>     improving the mapping performance).
> 
> 2. When Python sees an import request, it scans
> sys.path and
>     creates hook objects for each entry which it then
>     calls to say "go look and check whether you have
>     module X" until one of the hooks succeeds.
> 
> 3. Python then uses the hook object to complete the
> import
>     in much a similar way as e.g. SAX parsers call out
>     to event handlers.

Except for where the policy that associates a
"hook object" with an element of sys.path resides,
this is what iu.py does. In your case, you've got it
in the manager. In iu.py, each importer is given
a chance, until one says "yes, I can handle this".

I base this design choice on the observation that
managers, no matter how smart they are to start
with, tend to become very stupid over time <wink>.

-- Gordon
http://www.mcmillan-inc.com/



From Paul.Moore@atosorigin.com  Thu Dec  5 14:43:23 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Thu, 5 Dec 2002 14:43:23 -0000
Subject: [Python-Dev] (no subject)
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E61@UKDCX001.uk.int.atosorigin.com>

> > Guido van Rossum wrote:
> >=20
> > > Why don't you care about the backwards incompatibilities?
> >=20
> > Because it's addressed by using a str subclass.
>=20
> Which strikes *me* as an ugly hack. :-(

And it's dangerous. Code which assumes strings on sys.path and
modifies sys.path will fail silently rather than giving an error.

Example (not too realistic, I admit):

    sys.path =3D [ os.path.abspath(d) for d in sys.path ]

which collapses string subclasses to strings :-(

Paul.


From just@letterror.com  Thu Dec  5 14:50:37 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 15:50:37 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <3DEF126D.31051.677E3E3B@localhost>
Message-ID: <r01050400-1022-F18BA5E4086011D7AD06003065D5E7E4@[10.0.0.23]>

Gordon McMillan wrote:

> Here's what iu.py does:
>  
> The first time an entry on sys.path requires
> examination, the list of importers is traversed
> until one accepts it. That importer is associated
> with the name, so subsequent imports are only
> one dict lookup away from the importer.
> 
> Search order is as expected (sys.path.insert(0, ...)
> works). If your app only uses stuff from sys.path[0],
> no time is wasted finding the right importers for
> sys.path[1:].  

Ok, that's is a much clearer picture than I ever got from reading the code,
thanks. (And sorry Paul, you've been saying this all along; it's been an
entertaining and very educating trip for me ;-)

At the cost of some complexity this indeed solves all our problems:
- importers can be associated with sys.path elements matching a pattern
- adds a high level import hook mechanism
- makes plugging in a zipimporter a snap
- doesn't muck with sys.path

I'll have a whack at integrating this in import.c.

Just


From martin@v.loewis.de  Thu Dec  5 14:50:35 2002
From: martin@v.loewis.de (Martin v. Löwis)
Date: Thu, 5 Dec 2002 15:50:35 +0100
Subject: [Python-Dev] (no subject)
References: <16E1010E4581B049ABC51D4975CEDB885E2E61@UKDCX001.uk.int.atosorigin.com>
Message-ID: <002801c29c6d$ab8f2a90$aa2ce8d9@mira>

> Example (not too realistic, I admit):
>
>    sys.path = [ os.path.abspath(d) for d in sys.path ]
>
> which collapses string subclasses to strings :-(

Of course, for zipfile imports, this won't be a problem, since the
zipimport string is restored when the next import statement is executed.

Regards,
Martin



From fredrik@pythonware.com  Thu Dec  5 14:50:53 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 15:50:53 +0100
Subject: [Python-Dev] (no subject)
References: <16E1010E4581B049ABC51D4975CEDB885E2E61@UKDCX001.uk.int.atosorigin.com>
Message-ID: <1a8f01c29c6d$b694ab40$0900a8c0@spiff>

Paul Moore wrote:

> > > Guido van Rossum wrote:
> > >=20
> > > > Why don't you care about the backwards incompatibilities?
> > >=20
> > > Because it's addressed by using a str subclass.
> >=20
> > Which strikes *me* as an ugly hack. :-(
>=20
> And it's dangerous. Code which assumes strings on sys.path and
> modifies sys.path will fail silently rather than giving an error.

ignoring for a moment that your example do work as expected with the
proposed patch, since when is an "ImportError" a silent failure?

next example, please.

</F>



From pedronis@bluewin.ch  Thu Dec  5 14:41:07 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Thu, 5 Dec 2002 15:41:07 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-F18BA5E4086011D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <027b01c29c6c$58d74860$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> Gordon McMillan wrote:
>
> > Here's what iu.py does:
> >
> > The first time an entry on sys.path requires
> > examination, the list of importers is traversed
> > until one accepts it. That importer is associated
> > with the name, so subsequent imports are only
> > one dict lookup away from the importer.
> >
> > Search order is as expected (sys.path.insert(0, ...)
> > works). If your app only uses stuff from sys.path[0],
> > no time is wasted finding the right importers for
> > sys.path[1:].
>
> Ok, that's is a much clearer picture than I ever got from reading the code,
> thanks. (And sorry Paul, you've been saying this all along; it's been an
> entertaining and very educating trip for me ;-)
>
> At the cost of some complexity this indeed solves all our problems:
> - importers can be associated with sys.path elements matching a pattern
> - adds a high level import hook mechanism
> - makes plugging in a zipimporter a snap
> - doesn't muck with sys.path
>
> I'll have a whack at integrating this in import.c.

what about __path__'s role in this picture (btw this is not a rhetoric
question)?

regards.



From just@letterror.com  Thu Dec  5 15:12:01 2002
From: just@letterror.com (Just van Rossum)
Date: Thu,  5 Dec 2002 16:12:01 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <027b01c29c6c$58d74860$6d94fea9@newmexico>
Message-ID: <r01050400-1022-EBD58D18086311D7AD06003065D5E7E4@[10.0.0.23]>

Samuele Pedroni wrote:

> what about __path__'s role in this picture (btw this is not a rhetoric
> question)?

pkg.__path__ is just another sys.path. Whatever rules we make up for sys.path,
they are identical for pkg.__path__. When traversing path, find_module() doesn't
even know whether it's dealing with sys.path or a package's __path__.

The way I see it, the dict that maps path items to handlers also maps __path__
items; all handlers will live there, and there's just one such dict. I think
that a zipimporter would put something like this in __path__

  ["/path/to/archive.zip!packagename"]

So the zip handler will have to do a little more than just checking for a *.zip
extension. I assume you do something like it in Jython, too? (I picked up the
'!' from you I think; it makes it easier/quicker to check for zipness.)

Just


From bbum@codefab.com  Thu Dec  5 14:18:45 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Thu, 5 Dec 2002 09:18:45 -0500
Subject: [Python-Dev] Extending xmlrpclib to handle HTTP proxies
In-Reply-To: <165301c29c4a$c9b48880$0900a8c0@spiff>
Message-ID: <76A58EC5-085C-11D7-B650-000393877AE4@codefab.com>

On Thursday, December 5, 2002, at 05:31 AM, Fredrik Lundh wrote:
> mal wrote:
>> Doesn't urllib already support this out of the box ? You can use
>> its machinery for finding out the proxy settings and then add them
>> to Transport as needed.
>
> urllib doesn't allow you to override content-types and other things
> that you need to talk proper XML-RPC, and my "request body" [1]
> patch was rejected by GvR.

Yes -- I was surprised to find that urllib doesn't have the ability to 
return the prepared, but no data sent, HTTP instance.   This would seem 
to be very useful.   It would also seem to be easy to do -- split the 
open_http() method into two methods at the 'h = httplib.HTTP(host)' 
line.  The first method would simply return the created HTTP object 
while the second method would send the request, etc...

In any case...

> I'm pretty sure you can use urllib2 for this purpose, though, but I
> haven't looked deeper into that (mostly because people has been
> promising me proxy patches for years...)

... urllib2 seems to be the correct answer for a number of other 
reasons.   In particular, if the implementation is done correctly the 
developer can easily add a Handler to deal with any proxy/communication 
situations that I have not yet imagined (off the top of my head, 
dealing with TCP/IP proxying firewalls...).

Well -- I have pretty much coded myself into a corner out of which the 
only path is to add working proxy support to xmlrpclib.   Our product 
[see www.codefab.com, if curious] includes a first class python based 
XML-RPC client [open source'd soon].  Given that product is aimed at 
the corporate world (among others), the client must be able to deal 
with any existing infrastructure, including HTTP proxies that will so 
often be encountered in such situations.

> (Bill, note that Python CVS isn't the "master repository" for 
> xmlrpclib;
> if you come up with a clean and nice urllib2-based proxy transport,
> you can just mail it to me)

Will do.

> </F>
>
> 1) http://effbot.org/zone/python-urllib-requestbody.htm

thanks,
b.bum



From pedronis@bluewin.ch  Thu Dec  5 15:27:33 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Thu, 5 Dec 2002 16:27:33 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-EBD58D18086311D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <02bb01c29c72$d59df960$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
>
>   ["/path/to/archive.zip!packagename"]
>
> So the zip handler will have to do a little more than just checking for a
*.zip
> extension. I assume you do something like it in Jython, too? (I picked up the
> '!' from you I think; it makes it easier/quicker to check for zipness.)

yes at some point we used that, but then we switched to '/' on the other hand
the extesion .zip (or .jar in our case) is mandatory.



From Wiktor Sadowski" <art@wiktorsadowski.com  Wed Dec  4 16:15:15 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Wed, 4 Dec 2002 17:15:15 +0100
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <04b401c29bb0$5559e7e0$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_04B1_01C29BB8.B665AE40
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

I think we shouldn't use sys.path for zip archives.
Why not "copy" importing_modules_from_dlls(pyds) mechanism?

Then an archive.zip  would be imported if  found in ARCHIVES directory
or explicitly imported from  __init__.py (obviously:archive.zip  should =
be in the same directory)

Wiktor Sadowski
art@wiktorsadowski.com


------=_NextPart_000_04B1_01C29BB8.B665AE40
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV>I think we shouldn't use sys.path for zip archives.</DIV>
<DIV>Why not "copy" importing_modules_from_dlls(pyds) mechanism?</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Then an archive.zip&nbsp;&nbsp;would be imported if&nbsp; =
found&nbsp;in=20
ARCHIVES directory</DIV>
<DIV>or explicitly&nbsp;imported from &nbsp;__init__.py =
(obviously:archive.zip=20
&nbsp;should be in the same directory)</DIV>
<DIV>&nbsp;</DIV>
<DIV>Wiktor&nbsp;Sadowski</DIV>
<DIV><A =
href=3D"mailto:art@wiktorsadowski.com">art@wiktorsadowski.com</A></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV></BODY></HTML>

------=_NextPart_000_04B1_01C29BB8.B665AE40--



From =?windows-1250?Q?Martin_v._L=F6wis?=  Thu Dec  5 16:12:52 2002
From: =?windows-1250?Q?Martin_v._L=F6wis?= (=?windows-1250?Q?Martin_v._L=F6wis?=)
Date: Thu, 5 Dec 2002 17:12:52 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <04b401c29bb0$5559e7e0$0201010a@wiktor>
Message-ID: <007901c29c79$2a2fb260$aa2ce8d9@mira>

> I think we shouldn't use sys.path for zip archives.

Why do you think so?

> Why not "copy" importing_modules_from_dlls(pyds) mechanism?

What do you mean by that? What is "importing_modules_from_dlls"?

If you mean "one .pyd file is one Python module", then this approach
won't work for .zip files: People want many .py modules in a single .zip
file.

Regards,
Martin



From bbum@codefab.com  Thu Dec  5 17:13:27 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Thu, 5 Dec 2002 12:13:27 -0500
Subject: [Python-Dev] xmlrpclib + proxy support followup
Message-ID: <DE8100B6-0874-11D7-B650-000393877AE4@codefab.com>

</F>,

     I'm in the midst of implementing the Proxy support in xmlrpclib's 
Transport pattern and have a [minor] patch that seems to be necessary 
to my implementation.  I say 'seems' because I'm not that far in yet 
and may discover something else that makes this particular path of 
implementation invalid [but I don't think so].

     In particular, my goal is to implement a subclass of Transport that 
leverages the capabilities of urllib2 to provide Proxy support, HTTPs 
support, etc.. in a single class.   The goal is to provide the 
implementation entirely through the methods provided for creating and 
sending the request.  That is, through make_connection(), 
send_request(), send_host(), send_user_agent(), and send_content().

     However, I need a sixth method -- get_reply().

     I patched the Transport class in xmlrpclib as such:

     def request(self, host, handler, request_body, verbose=0):
         # issue XML-RPC request
         ....
         errcode, errmsg, headers = self.get_reply(h)

     ....

     def get_reply(self, connection):
         return connection.getReply()

	This would seem to be a trivial change to the existing xmlrpclib that 
greatly increases the ability to modify its behavior via subclassing, 
etc...

     Onwards,
     b.bum
  
    



From fredrik@pythonware.com  Thu Dec  5 17:23:56 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 18:23:56 +0100
Subject: [Python-Dev] Re: xmlrpclib + proxy support followup
References: <DE8100B6-0874-11D7-B650-000393877AE4@codefab.com>
Message-ID: <00c201c29c83$18dd99a0$ced241d5@hagrid>

bill wrote:

>      However, I need a sixth method -- get_reply().
> 
>      I patched the Transport class in xmlrpclib as such:
> 
>      def request(self, host, handler, request_body, verbose=0):
>          # issue XML-RPC request
>          ....
>          errcode, errmsg, headers = self.get_reply(h)
> 
>      ....
> 
>      def get_reply(self, connection):
>          return connection.getReply()
> 
> This would seem to be a trivial change to the existing xmlrpclib that 
> greatly increases the ability to modify its behavior via subclassing, 
> etc...

+1

(and feel free to add self.get_file(h) too, if you need that).

</F>



From gmcm@hypernet.com  Thu Dec  5 17:01:47 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Thu, 5 Dec 2002 12:01:47 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <027b01c29c6c$58d74860$6d94fea9@newmexico>
Message-ID: <3DEF402B.782.6830EE86@localhost>

On 5 Dec 2002 at 15:41, Samuele Pedroni wrote:

> > Gordon McMillan wrote:
> >
> > > Here's what iu.py does:

[...]

> what about __path__'s role in this picture (btw this
> is not a rhetoric question)? 

In iu, __path__ is (by default) treated just like
a "local" sys.path. In fact, you *could* say that
sys.path is just the __path__ of the package-with-no-name.

BTW, iu does allow non string objects in sys.path.
If it exposes "getmod(name)", that will be tried
in preference to iu's own machinery.

-- Gordon
http://www.mcmillan-inc.com/



From Wiktor Sadowski" <art@wiktorsadowski.com  Wed Dec  4 19:25:24 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Wed, 4 Dec 2002 20:25:24 +0100
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <074b01c29bca$e5298dc0$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_0748_01C29BD3.465E8700
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

> I think we shouldn't use sys.path for zip archives.
>Why do you think so?
IMO:Why make changes if it's not necessary .?

> Why not "copy" importing_modules_from_dlls(pyds) mechanism?
>What do you mean by that? What is "importing_modules_from_dlls"?

Should be "importing_module_from_dlls".

>If you mean "one .pyd file is one Python module", then this approach
>won't work for .zip files:People want many .py modules in a single .zip
>file.

It will work if handled by the zip importer.

As for ARCHIVES directory: a "virtual" path for all zip files could be =
built for lookup.
Importing from __init__.py could be done explicitly with:

from MyArchive_with_many_modules import MyModule
or
import MyArchive_with_many_modules.MyModule

where  MyArchive_with_many_modules is MyArchive_with_many_modules.zip=20
in the same directory (or in ARCHIVES)

Regards,
Wiktor=20












------=_NextPart_000_0748_01C29BD3.465E8700
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV>&gt; I think we shouldn't use sys.path for zip archives.<BR>&gt;Why =
do you=20
think so?</DIV>
<DIV>IMO:Why make changes if&nbsp;it's not necessary .?</DIV>
<DIV><FONT face=3DArial size=3D2><FONT face=3D"Times New Roman" =
size=3D3><FONT=20
face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>&gt; Why not "copy" importing_modules_from_dlls(pyds)=20
mechanism?<BR>&gt;What do you mean by that? What is=20
"importing_modules_from_dlls"?</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Should be "importing_module_from_dlls".</DIV>
<DIV><BR>&gt;If you mean "one .pyd file is one Python module", then this =

approach<BR>&gt;won't work for .zip files:People want many .py modules =
in a=20
single .zip<BR>&gt;file.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>It will work if handled by the zip importer.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>As for ARCHIVES directory: a "virtual" path for all zip files could =
be=20
built for lookup.</DIV>
<DIV>Importing from __init__.py could be done explicitly with:</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>from MyArchive_with_many_modules import MyModule</DIV>
<DIV>or</DIV>
<DIV>import&nbsp;MyArchive_with_many_modules.MyModule</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>where&nbsp;&nbsp;MyArchive_with_many_modules is=20
MyArchive_with_many_modules.zip </DIV>
<DIV>in the same directory (or in ARCHIVES)</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Regards,</DIV>
<DIV>Wiktor&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><BR><BR></FONT><BR></DIV></FONT></BODY></HTML>

------=_NextPart_000_0748_01C29BD3.465E8700--



From niemeyer@conectiva.com  Thu Dec  5 19:30:53 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 17:30:53 -0200
Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30
In-Reply-To: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU>
Message-ID: <20021205173053.A18494@ibook.distro.conectiva>

> ================================
> `Dictionary evaluation order`__
> ================================
> __ http://mail.python.org/pipermail/python-dev/2002-November/030458.html
> 
[...]
> Finn Bock said that Jython went f1, f2, f3, f4.  In that case Guido didn't
> see any reason to fix it.  But Tim Peters brought up the point that the
[...]

I belive the correct sentence here would be:

"In that case Guido didn't see any reason to *block* the fix."

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From =?windows-1250?Q?Martin_v._L=F6wis?=  Thu Dec  5 20:01:25 2002
From: =?windows-1250?Q?Martin_v._L=F6wis?= (=?windows-1250?Q?Martin_v._L=F6wis?=)
Date: Thu, 5 Dec 2002 21:01:25 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <074b01c29bca$e5298dc0$0201010a@wiktor>
Message-ID: <016f01c29c99$18967c80$aa2ce8d9@mira>

>> Why not "copy" importing_modules_from_dlls(pyds) mechanism?
>>What do you mean by that? What is "importing_modules_from_dlls"?

>Should be "importing_module_from_dlls".

I still don't understand. Is that some feature of Python?

>>If you mean "one .pyd file is one Python module", then this approach
>>won't work for .zip files:People want many .py modules in a single
.zip
>>file.

>It will work if handled by the zip importer.

How will the zip importer know what zipfile to read?

If I have "python23.zip", and do

import httplib

what mechanism will extract httplib.py from python23.zip?

> from MyArchive_with_many_modules import MyModule

So you could only zip entire packages? This is insufficient.

Regards,
Martin



From niemeyer@conectiva.com  Thu Dec  5 21:26:16 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 19:26:16 -0200
Subject: [Python-Dev] Another approach for the import mechanism
Message-ID: <20021205192616.B18494@ibook.distro.conectiva>

I propose a different approach to the importer hook mechanism:

- An importer register itself with sys.register_importer(), as suggested
  by MAL.

- No .zip/.tar/.whatever files are ever included in sys.path

- Alternative importers are only considered when the default importer
  mechanism fails (each entry in the path at a time, so that
  precedence is preserved).

- Alternative importers, when activated, would be given the module name
  being imported, and would look for entries in the "current" iterated
  path by themselves.

- A zip importer would, for example, look if the current iterated path
  have a file named "__init__.zip".

- The same importer could also be able to look for a file named
  "mypackage.zip", if "import mypackage.foo" is tried.

What's your opinion about that?

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From sholden@holdenweb.com  Thu Dec  5 21:28:25 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Thu, 5 Dec 2002 16:28:25 -0500
Subject: [Python-Dev] New and Improved Import Hooks
References: <074b01c29bca$e5298dc0$0201010a@wiktor> <016f01c29c99$18967c80$aa2ce8d9@mira>
Message-ID: <016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com>

[MvL]
> >> Why not "copy" importing_modules_from_dlls(pyds) mechanism?
> >>What do you mean by that? What is "importing_modules_from_dlls"?
>
[Wiktor]
> >Should be "importing_module_from_dlls".
>
> I still don't understand. Is that some feature of Python?
>
I think Wiktor is referring to extensions, which under Windows are often
provided as .pyd DLLs.

> >>If you mean "one .pyd file is one Python module", then this approach
> >>won't work for .zip files:People want many .py modules in a single
> .zip
> >>file.
>
> >It will work if handled by the zip importer.
>
> How will the zip importer know what zipfile to read?
>
> If I have "python23.zip", and do
>
> import httplib
>
> what mechanism will extract httplib.py from python23.zip?
>
> > from MyArchive_with_many_modules import MyModule
>
> So you could only zip entire packages? This is insufficient.
>
I agree. What I'd like to see is the ability to replace a directory on
sys.path by a reference to a zipped copy, including any subdirectories, and
have all imports work just as they did before (whether they be extensions or
pure Python modules].

A quick look at my own code base, admittedly puny, reveals no statements
that would break were non-strings to be added to sys.path. Is there some way
to trap assignments to sys.path so a warning could be issued if sufficiently
un-stringy elements were discovered in it? Clearly it isn't enough to trap
assignment to and insertion of elements if breakage is to be totaly avoided.
We  have to know when the name is assigned-to as well.

regards
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Previous .sig file retired to                    www.homeforoldsigs.com
-----------------------------------------------------------------------





From sholden@holdenweb.com  Thu Dec  5 21:35:52 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Thu, 5 Dec 2002 16:35:52 -0500
Subject: [Python-Dev] Another approach for the import mechanism
References: <20021205192616.B18494@ibook.distro.conectiva>
Message-ID: <017701c29ca6$49a74b80$7101a8c0@holdenweb.com>

> I propose a different approach to the importer hook mechanism:
>
> - An importer register itself with sys.register_importer(), as suggested
>   by MAL.
>
> - No .zip/.tar/.whatever files are ever included in sys.path
>
> - Alternative importers are only considered when the default importer
>   mechanism fails (each entry in the path at a time, so that
>   precedence is preserved).
>
Far too restrictive. You seem to be saying (unless I misunderstand) that an
alternative import can't override an element on the traditional sys.path. Is
there to be no way to replace existing system components by providing
replacements in zip files, in the same way that I could currently replace
httplib by putting my own httplib in a directory which I then place at the
beginning of sys.path?

> - Alternative importers, when activated, would be given the module name
>   being imported, and would look for entries in the "current" iterated
>   path by themselves.
>
Huh? Why restrict alternative importers to using a path? What if my importer
wants to provide randomly generated byte codes as part of a genetic
programming experiment?

> - A zip importer would, for example, look if the current iterated path
>   have a file named "__init__.zip".
>
...and I'm not really sure what you mean by "iterated path".

> - The same importer could also be able to look for a file named
>   "mypackage.zip", if "import mypackage.foo" is tried.
>
> What's your opinion about that?
>
I think I must be misunderstanding at a fairly basic level. This would seem
pretty inflexible to me.

regards
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Previous .sig file retired to                    www.homeforoldsigs.com
-----------------------------------------------------------------------


> http://mail.python.org/mailman/listinfo/python-dev
>
>




From niemeyer@conectiva.com  Thu Dec  5 21:41:18 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 19:41:18 -0200
Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS
In-Reply-To: <asip5f$9uf$1@main.gmane.org>
References: <15852.57674.16727.930806@montanaro.dyndns.org> <asip5f$9uf$1@main.gmane.org>
Message-ID: <20021205194118.C18494@ibook.distro.conectiva>

> Feel free to only edit Misc/NEWS; before releases I'll go through it
> to find things I've missed, so there's no need to update whatsnew23.tex.
> I don't mind if people edit whatsnew23.tex to add items, but it's 
> certainly not necessary.

Thanks for mentioning that. I was really wondering about what to do.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From niemeyer@conectiva.com  Thu Dec  5 21:58:55 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 19:58:55 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <017701c29ca6$49a74b80$7101a8c0@holdenweb.com>
References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com>
Message-ID: <20021205195855.A19843@ibook.distro.conectiva>

> > - Alternative importers are only considered when the default importer
> >   mechanism fails (each entry in the path at a time, so that
> >   precedence is preserved).
>
> Far too restrictive. You seem to be saying (unless I misunderstand)
> that an alternative import can't override an element on the
> traditional sys.path.

Why not? I belive it's exactly the other way around. You have
more possibilities with that scheme.

> Is there to be no way to replace existing system components by
> providing replacements in zip files, in the same way that I could
> currently replace httplib by putting my own httplib in a directory
> which I then place at the beginning of sys.path?

I'm not sure if I understood what you're trying to say. But if I
understood that correctly, there are even more possibilities:

Having one of:

/some/dir/httplib/__init__.zip
/some/dir/httplib.zip

You'd do:

sys.path.insert(0, "/some/dir")

> > - Alternative importers, when activated, would be given the module name
> >   being imported, and would look for entries in the "current" iterated
> >   path by themselves.
>
> Huh? Why restrict alternative importers to using a path? What if my
> importer wants to provide randomly generated byte codes as part of a
> genetic programming experiment?

Then you'd have to work with ihooks. IIRC, none of the proposed
mechanisms would allow that.

Also, I wouldn't say that "randomly generated byte codes" should be
*imported*. Just compile it.

> > - A zip importer would, for example, look if the current iterated path
> >   have a file named "__init__.zip".
>
> ...and I'm not really sure what you mean by "iterated path".

for path in sys.path:
	import_default(path)
	import_alternate(path)

> I think I must be misunderstanding at a fairly basic level. This would
> seem pretty inflexible to me.

Does it look better now?

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From fredrik@pythonware.com  Thu Dec  5 22:01:18 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 5 Dec 2002 23:01:18 +0100
Subject: [Python-Dev] Another approach for the import mechanism
References: <20021205192616.B18494@ibook.distro.conectiva>
Message-ID: <005701c29ca9$d8517ba0$ced241d5@hagrid>

Gustavo Niemeyer wrote:

> What's your opinion about that?

Who's going to call sys.register_importer()?

What does "current iterated path" mean?

How do I create a ZIP-only distribution?

Do I need to restructure my code to use custom importers, or am
I missing something?  If I want to ship multiple ZIP archives, do I
really have to name each one of them __init__.zip and place each
one in a different directory?

Have you tried Just's patch?

What problem does your solution solve that his code doesn't already
handle?

Etc.

</F>



From pedronis@bluewin.ch  Thu Dec  5 21:54:11 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Thu, 5 Dec 2002 22:54:11 +0100
Subject: [Python-Dev] Another approach for the import mechanism
References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva>
Message-ID: <06e601c29ca8$d8a31380$6d94fea9@newmexico>

From: "Gustavo Niemeyer" <niemeyer@conectiva.com>
> Having one of:
>
> /some/dir/httplib/__init__.zip
> /some/dir/httplib.zip
>
> You'd do:
>
> sys.path.insert(0, "/some/dir")
>

it's confusing (and imcompatible wrt to how Java/Jython do things now).

__init__.py exists so that a package has a first-class counterpart, namely a
first class-module.

Zipfiles should be able to embrace more than just one package and should be
transparent.



From skip@pobox.com  Thu Dec  5 22:09:34 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 5 Dec 2002 16:09:34 -0600
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021205195855.A19843@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <017701c29ca6$49a74b80$7101a8c0@holdenweb.com>
 <20021205195855.A19843@ibook.distro.conectiva>
Message-ID: <15855.52894.524073.885235@montanaro.dyndns.org>

    Gustavo> Having one of:

    Gustavo> /some/dir/httplib/__init__.zip
    Gustavo> /some/dir/httplib.zip

    Gustavo> You'd do:

    Gustavo> sys.path.insert(0, "/some/dir")

Remember, a zip file looks like a directory because of its embedded
structure.  If you had httplib.zip it would have to contain at least one .py
or .so (or .pyd or ...) file.  I think you would insert
/some/dir/httplib.zip into sys.path.

-- 
Skip Montanaro - skip@pobox.com
http://www.mojam.com/
http://www.musi-cal.com/


From martin@v.loewis.de  Thu Dec  5 22:14:50 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 05 Dec 2002 23:14:50 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021205192616.B18494@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
Message-ID: <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de>

Gustavo Niemeyer <niemeyer@conectiva.com> writes:

> - No .zip/.tar/.whatever files are ever included in sys.path
[...]
> What's your opinion about that?

It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they
MUST be considered in order with all other items in PYTHONPATH;
the order requirement also applies for sys.path.

Regards,
Martin



From bbum@codefab.com  Thu Dec  5 22:24:19 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Thu, 5 Dec 2002 17:24:19 -0500
Subject: [Python-Dev] urllib2.build_opener() broken?
Message-ID: <4C444805-08A0-11D7-B650-000393877AE4@codefab.com>

I'm invoking build_opener() as...

         opener = urllib2.build_opener(proxyHandler)

.... and the handlers appear to be out of order-- the ProxyHandler that 
is supposed to be at the beginning is not ...

[<urllib2.UnknownHandler instance at 0xae6b00>, <urllib2.HTTPHandler 
instance at 0xae6cb0>, <urllib2.HTTPDefaultErrorHandler instance at 
0xae6dd0>, <urllib2.HTTPRedirectHandler instance at 0xae6f60>, 
<urllib2.FTPHandler instance at 0xae6f90>, <urllib2.FileHandler 
instance at 0xae7230>, <urllib2.HTTPSHandler instance at 0xae7320>, 
<urllib2.ProxyHandler instance at 0xae6270>]

 From the comments on build_opener(), it would seem that it doesn't take 
this into account.  However, if it doesn't, then the example used in 
the documentation at the top of urllib2 could not possibly work.

Whose broken?  The module or me?

thanks,
b.bum



From skip@pobox.com  Thu Dec  5 22:24:55 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 5 Dec 2002 16:24:55 -0600
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <15855.52894.524073.885235@montanaro.dyndns.org>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <017701c29ca6$49a74b80$7101a8c0@holdenweb.com>
 <20021205195855.A19843@ibook.distro.conectiva>
 <15855.52894.524073.885235@montanaro.dyndns.org>
Message-ID: <15855.53815.906868.938394@montanaro.dyndns.org>

    Skip> Remember, a zip file looks like a directory because of its
    Skip> embedded structure.  If you had httplib.zip it would have to
    Skip> contain at least one .py or .so (or .pyd or ...) file.  I think
    Skip> you would insert /some/dir/httplib.zip into sys.path.

Just rereading my own note, I realize you might have components after the
zip file name.  Suppose you had web.zip which contained directories http,
gopher, ftp, and nntp, each of which was itself a package (had a __init__.py
file).  You might modify sys.path like so:

    newdirs = [os.path.join("/some/dir/web.zip", x)
                  for x in "http gopher ftp nttp".split()]
    sys.path.extend(newdirs)

Skip


From bbum@codefab.com  Thu Dec  5 22:27:08 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Thu, 5 Dec 2002 17:27:08 -0500
Subject: [Python-Dev] build_opener() broken
Message-ID: <B094D494-08A0-11D7-B650-000393877AE4@codefab.com>

It is broken, not me.

See bug #487471.

https://sourceforge.net/tracker/ 
?group_id=5470&atid=105470&func=detail&aid=487471

Guess I have to fix that, too.

b.bum



From niemeyer@conectiva.com  Thu Dec  5 22:31:37 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 20:31:37 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <005701c29ca9$d8517ba0$ced241d5@hagrid>
References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid>
Message-ID: <20021205203137.C19843@ibook.distro.conectiva>

> > What's your opinion about that?
> 
> Who's going to call sys.register_importer()?

Sorry, but I'll answer your question with another question:

Who's going to notify to the importer mechanism that zip importers
are available, in the current implementation?

> What does "current iterated path" mean?

for path in sys.path:
	default_importer(path)
	alternative_importer(path)

> How do I create a ZIP-only distribution?

Include __init__.zip in /usr/lib/python2.2.

> Do I need to restructure my code to use custom importers, or am
> I missing something?

Why would you need to?

> If I want to ship multiple ZIP archives, do I really have to name each
> one of them __init__.zip and place each one in a different directory?

It depends on what your zip archives are. If you want to include their
content as "top level" modules (as the standard library), yes. If they
are packages by themselves (like the email package), no.

> Have you tried Just's patch?

Do you agree that this is a different approach?

> What problem does your solution solve that his code doesn't already
> handle?

- Avoids including zip files in the path.

- Allows one to have zip packages inside other packages.

- Allows me to ship a package inside a zip file, without asking
  the user to change his path.

- Allows me to compress a single file (foobar.py.bz2).

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From niemeyer@conectiva.com  Thu Dec  5 22:33:12 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 20:33:12 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de>
References: <20021205192616.B18494@ibook.distro.conectiva> <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021205203312.A20315@ibook.distro.conectiva>

> It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they

Why? Have you promised that to someone? :-))

> MUST be considered in order with all other items in PYTHONPATH;
> the order requirement also applies for sys.path.

I told that would be onored (perhaps in an uncomprehensible way). Each
path is tried with the alternative importers once the default importer
has failed on *that* specific path.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From martin@v.loewis.de  Thu Dec  5 22:38:50 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 05 Dec 2002 23:38:50 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com>
References: <074b01c29bca$e5298dc0$0201010a@wiktor>
 <016f01c29c99$18967c80$aa2ce8d9@mira>
 <016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com>
Message-ID: <m3smxc6qh1.fsf@mira.informatik.hu-berlin.de>

"Steve Holden" <sholden@holdenweb.com> writes:

> > >Should be "importing_module_from_dlls".
> >
> > I still don't understand. Is that some feature of Python?
> >
> I think Wiktor is referring to extensions, which under Windows are often
> provided as .pyd DLLs.

Yes, but then I understand how that is related to zipfile imports: A
.pyd provides only a single module. A .zip file must supply many
modules.

> A quick look at my own code base, admittedly puny, reveals no statements
> that would break were non-strings to be added to sys.path. Is there some way
> to trap assignments to sys.path so a warning could be issued if sufficiently
> un-stringy elements were discovered in it? 

This isn't really possible, as this is just a dict
modification. You'ld also have to catch assignments to
sys.__dict__['path']

Regards,
Martin


From pedronis@bluewin.ch  Thu Dec  5 22:28:54 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Thu, 5 Dec 2002 23:28:54 +0100
Subject: [Python-Dev] Another approach for the import mechanism
Message-ID: <086b01c29cad$b1de3400$6d94fea9@newmexico>

From: "Gustavo Niemeyer" <niemeyer@conectiva.com>
> It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they
>
> Why? Have you promised that to someone? :-))

Jython has implented it that way because it was agreed that this was what PEP
273 was going to allow.

regards.



From niemeyer@conectiva.com  Thu Dec  5 22:44:01 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 20:44:01 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <15855.53815.906868.938394@montanaro.dyndns.org>
References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva> <15855.52894.524073.885235@montanaro.dyndns.org> <15855.53815.906868.938394@montanaro.dyndns.org>
Message-ID: <20021205204401.D19843@ibook.distro.conectiva>

> Just rereading my own note, I realize you might have components after the
> zip file name.  Suppose you had web.zip which contained directories http,
> gopher, ftp, and nntp, each of which was itself a package (had a __init__.py
> file).  You might modify sys.path like so:
> 
>     newdirs = [os.path.join("/some/dir/web.zip", x)
>                   for x in "http gopher ftp nttp".split()]
>     sys.path.extend(newdirs)

I'm not sure how useful that would be (i.e. I haven't done that by
myself), but that could be handled in the proposed mechanism as well. If
/some/dir/web/http/gopher doesn't exist, that mechanism could check if
http.zip exists, or web.zip exists (or __init__.zip inside those
directories). This also has the advantage that you don't have to enforce
a specific format in your code. If you later discover that tar.bz2 has a
better compression, just go for it. OTOH, I understand that this would
lead to extra tests in comparison with your example.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From niemeyer@conectiva.com  Thu Dec  5 22:52:38 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 20:52:38 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <06e601c29ca8$d8a31380$6d94fea9@newmexico>
References: <20021205192616.B18494@ibook.distro.conectiva> <017701c29ca6$49a74b80$7101a8c0@holdenweb.com> <20021205195855.A19843@ibook.distro.conectiva> <06e601c29ca8$d8a31380$6d94fea9@newmexico>
Message-ID: <20021205205238.E19843@ibook.distro.conectiva>

> it's confusing (and imcompatible wrt to how Java/Jython do things now).

Sorry. I wouldn't like to do something in a different way than Jython
does, or to obligate you to rework that. OTOH, I belive that this is a
better scheme, and my obligation is to discuss that with you. If it's of
common sense that this isn't good, it won't be accepted, and that's all.

> __init__.py exists so that a package has a first-class counterpart,
> namely a first class-module.

That's a very similar idea. But if you don't like __init__.zip, just
rename it to __module__.zip, or whatever.

> Zipfiles should be able to embrace more than just one package and
> should be transparent.

Can you please explain how the propose changes that?

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From niemeyer@conectiva.com  Thu Dec  5 22:56:21 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 20:56:21 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <086b01c29cad$b1de3400$6d94fea9@newmexico>
References: <086b01c29cad$b1de3400$6d94fea9@newmexico>
Message-ID: <20021205205621.F19843@ibook.distro.conectiva>

> > Why? Have you promised that to someone? :-))
> 
> Jython has implented it that way because it was agreed that this was what PEP
> 273 was going to allow.

[replicated]

Sorry. I wouldn't like to do something in a different way than Jython
does, or to obligate you (or someone) to rework that. OTOH, I belive
that this is a better scheme, and my obligation is to discuss that with
you. If it's of common sense that this isn't good, it won't be accepted,
and that's all.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From pedronis@bluewin.ch  Thu Dec  5 22:45:47 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Thu, 5 Dec 2002 23:45:47 +0100
Subject: [Python-Dev] Another approach for the import mechanism
References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva>
Message-ID: <09e901c29cb0$0da094c0$6d94fea9@newmexico>

> - Allows one to have zip packages inside other packages.

you can achieve this with __path__

>
> - Avoids including zip files in the path.

I want to support that.

>
> - Allows me to ship a package inside a zip file, without asking
>   the user to change his path.
>
> - Allows me to compress a single file (foobar.py.bz2).
>

Java has support for zipfiles from day one, from my user experience they are
both YAGNI.

sys.path content from my point of view is really a deployment time issue.

The 2nd point is not directly addressed by your proposal, and supporting it
would make for more complicated importer impls.

regards.



From Wiktor Sadowski" <art@wiktorsadowski.com  Wed Dec  4 23:24:31 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Thu, 5 Dec 2002 00:24:31 +0100
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <07b101c29bec$4c902f20$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_07AE_01C29BF4.AE00AAC0
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

>Should be "importing_module_from_dlls".

>I still don't understand. Is that some feature of Python?

Under Windows you can put a Module.pyd into Python/DLLs directory and:
import Module

>>If you mean "one .pyd file is one Python module", then this approach
>>won't work for .zip files:People want many .py modules in a single
.zip
>>file.

>It will work if handled by the zip importer.

>How will the zip importer know what zipfile to read?

>If I have "python23.zip", and do

>import httplib

>what mechanism will extract httplib.py from python23.zip?

As I said: a virtual directory-tree could be built for all zip files in=20
ARCHIVES directory:

ZipArchives:
   ZipArchives/python23.zip:
                ZipArchives/python23.zip/Lib:
                    ZipArchives/python23.zip/Lib/httplib.pyc
   ZipArchives/another.zip:=20
               ZipArchives/another.zip/another_directory:
                       =
ZipArchives/another.zip/another_directory/another_module.pyc     =20

> from MyArchive_with_many_modules import MyModule

>So you could only zip entire packages?=20
No.
"from MyArchive import MyModule"
would be resolved by zip importer as:
if there is MyModule in MyArchive.zip then then import MyModule
(means extract and import)

and
import Module:
if Module.zip contains Module import Module

Wiktor


------=_NextPart_000_07AE_01C29BF4.AE00AAC0
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV>&gt;Should be "importing_module_from_dlls".<BR><BR>&gt;I still =
don't=20
understand. Is that some feature of Python?</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Under Windows you can put a Module.pyd into Python/DLLs directory=20
and:</DIV>
<DIV><FONT face=3DArial size=3D2><FONT face=3D"Times New Roman" =
size=3D3>import=20
Module</FONT></DIV>
<DIV><BR><FONT face=3D"Times New Roman" size=3D3>&gt;&gt;If you mean =
"one .pyd file=20
is one Python module", then this approach<BR>&gt;&gt;won't work for .zip =

files:People want many .py modules in a=20
single<BR>.zip<BR>&gt;&gt;file.<BR><BR>&gt;It will work if handled by =
the zip=20
importer.<BR><BR>&gt;How will the zip importer know what zipfile to=20
read?<BR><BR>&gt;If I have "python23.zip", and do<BR><BR>&gt;import=20
httplib<BR><BR>&gt;what mechanism will extract httplib.py from=20
python23.zip?</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3></FONT>&nbsp;</DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>As I said:&nbsp;a virtual=20
directory-tree could be built for all zip files in </FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>ARCHIVES =
directory:</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3></FONT>&nbsp;</DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>ZipArchives:</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>&nbsp;&nbsp;=20
ZipArchives/python23.zip:</FONT></DIV>
<DIV><FONT face=3D"Times New Roman"=20
size=3D3>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;=20
ZipArchives/python23.zip/Lib:</FONT></DIV>
<DIV><FONT face=3D"Times New Roman"=20
size=3D3>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=20
ZipArchives/python23.zip/Lib/httplib.pyc</FONT></DIV>
<DIV><FONT face=3D"Times New Roman"=20
size=3D3>&nbsp;&nbsp;&nbsp;ZipArchives/another.zip:&nbsp;</FONT></DIV>
<DIV><FONT face=3D"Times New Roman"=20
size=3D3>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;=20
ZipArchives/another.zip/another_directory:</FONT></DIV>
<DIV><FONT face=3D"Times New Roman"=20
size=3D3>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=20
ZipArchives/another.zip/another_directory/another_module.pyc&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;=20
</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3></FONT>&nbsp;</DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>&gt; from =
MyArchive_with_many_modules=20
import MyModule<BR><BR>&gt;So you could only zip entire packages? =
</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>No.</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>"from MyArchive import=20
MyModule"</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>would be resolved by zip =
importer=20
as:</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>if there is MyModule in =
MyArchive.zip=20
then then import MyModule</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>(means extract and =
import)</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3></FONT>&nbsp;</DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>and</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>import Module:</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3>if Module.zip contains =
Module import=20
Module</FONT></DIV>
<DIV><FONT face=3D"Times New Roman" size=3D3></FONT>&nbsp;</DIV>
<DIV><FONT face=3D"Times New Roman"=20
size=3D3>Wiktor</FONT><BR></DIV></FONT></BODY></HTML>

------=_NextPart_000_07AE_01C29BF4.AE00AAC0--



From martin@v.loewis.de  Thu Dec  5 23:05:18 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 00:05:18 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021205203312.A20315@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de>
 <20021205203312.A20315@ibook.distro.conectiva>
Message-ID: <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de>

Gustavo Niemeyer <niemeyer@conectiva.com> writes:

> > It's unacceptable. Zip files MUST be allowed in PYTHONPATH, and they
> 
> Why? Have you promised that to someone? :-))

Sure. See PEP 273. Also, it is so similar to the Java CLASSPATH
feature that it better be identical.


> > MUST be considered in order with all other items in PYTHONPATH;
> > the order requirement also applies for sys.path.
> 
> I told that would be onored (perhaps in an uncomprehensible way).

If so, it was indeed incomprehensible :-(

Rereading your proposal, it appears that you are also proposing that
you can only package up entire Python packages with your strategy.

It also is a requirement that you can zip up the Pythons standard
library.

Regards,
Martin



From martin@v.loewis.de  Thu Dec  5 23:17:44 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 00:17:44 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <07b101c29bec$4c902f20$0201010a@wiktor>
References: <07b101c29bec$4c902f20$0201010a@wiktor>
Message-ID: <m365u86oo7.fsf@mira.informatik.hu-berlin.de>

"Wiktor Sadowski" <art@wiktorsadowski.com> writes:

> >I still don't understand. Is that some feature of Python?
> 
> Under Windows you can put a Module.pyd into Python/DLLs directory and:
> import Module

I see. Notice that there is nothing special about the DLLs
directory. Instead, you can have .pyd files anywhere on sys.path, and
the DLLs directory is included in sys.path - that's why .pyd files are
found in the DLLs directory.

> >If I have "python23.zip", and do
> 
> >import httplib
[...]
> >So you could only zip entire packages? 
> No.

All your explanations indicate otherwise.

> "from MyArchive import MyModule"
> would be resolved by zip importer as:
> if there is MyModule in MyArchive.zip then then import MyModule
> (means extract and import)
> 
> and
> import Module:
> if Module.zip contains Module import Module

So if I do

import httplib

it will check httplib.zip. Correct? That won't help, since httplib is
in python23.zip, not in httplib.zip.

According to your proposal, I would have to write

from python23 import httplib

But I don't want to, since httplib is a toplevel module, which is not
included in a python23 package.

Regards,
Martin



From niemeyer@conectiva.com  Thu Dec  5 23:21:27 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 21:21:27 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <09e901c29cb0$0da094c0$6d94fea9@newmexico>
References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico>
Message-ID: <20021205212127.G19843@ibook.distro.conectiva>

> > - Allows one to have zip packages inside other packages.
> 
> you can achieve this with __path__

Indeed. Thanks for mentioning that.

> > - Avoids including zip files in the path.
> 
> I want to support that.

Well... :-)

> > - Allows me to ship a package inside a zip file, without asking
> >   the user to change his path.
> >
> > - Allows me to compress a single file (foobar.py.bz2).
> 
> Java has support for zipfiles from day one, from my user experience
> they are both YAGNI.

If I want to provide packaged RPMs for small systems, I'd like
that *every* module/package which can be installed optionally, be
packaged inside a compressed file. There are real usage for
that right now.

> sys.path content from my point of view is really a deployment time
> issue.

I'm not sure about what you mean here.

> The 2nd point is not directly addressed by your proposal, and supporting it
> would make for more complicated importer impls.

Yes, it is addressed. My propose is to allow the importer mechanism to
check if there's something importable with the given name in the given
path, instead of telling him what to import. With that in mind,
importing foobar.py.bz2 is just a matter of checking for
<modulename>.py.bz2 in the given path.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From niemeyer@conectiva.com  Thu Dec  5 23:26:04 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 5 Dec 2002 21:26:04 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de>
References: <20021205192616.B18494@ibook.distro.conectiva> <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de> <20021205203312.A20315@ibook.distro.conectiva> <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021205212604.H19843@ibook.distro.conectiva>

> If so, it was indeed incomprehensible :-(
> 
> Rereading your proposal, it appears that you are also proposing that
> you can only package up entire Python packages with your strategy.
> 
> It also is a requirement that you can zip up the Pythons standard
> library.

I need some english classes then. :-)

/usr/lib/python2.2/__init__.zip (or __module__.zip, or whatever)

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From martin@v.loewis.de  Thu Dec  5 23:33:41 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 00:33:41 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021205212127.G19843@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <005701c29ca9$d8517ba0$ced241d5@hagrid>
 <20021205203137.C19843@ibook.distro.conectiva>
 <09e901c29cb0$0da094c0$6d94fea9@newmexico>
 <20021205212127.G19843@ibook.distro.conectiva>
Message-ID: <m3vg2859d6.fsf@mira.informatik.hu-berlin.de>

Gustavo Niemeyer <niemeyer@conectiva.com> writes:

> Yes, it is addressed. My propose is to allow the importer mechanism to
> check if there's something importable with the given name in the given
> path, instead of telling him what to import. With that in mind,
> importing foobar.py.bz2 is just a matter of checking for
> <modulename>.py.bz2 in the given path.

It appears that this can't be really done with the proposed mechanism,
as you want to have a multiple hooks for a single source "URL".

If you *only* want to have a .py.bz2 importer, you could replace every
directory on sys.path with the pybz2importer, which would check for
.py.bz2 files in the directory, and, if that fails, calls
imp.find_module.

Of course, if somebody wants to provide a crypt importer (which
decrypts the source before importing it) in addition to the
pybz2importer, then you get the same coordination problem as with all
prior import hooks.

So you are back to calling all registered hooks for all items on
sys.path, which might be expensive.

Regards,
Martin



From martin@v.loewis.de  Thu Dec  5 23:37:55 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 00:37:55 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021205212604.H19843@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de>
 <20021205203312.A20315@ibook.distro.conectiva>
 <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de>
 <20021205212604.H19843@ibook.distro.conectiva>
Message-ID: <m3r8cw5964.fsf@mira.informatik.hu-berlin.de>

Gustavo Niemeyer <niemeyer@conectiva.com> writes:

> I need some english classes then. :-)
> 
> /usr/lib/python2.2/__init__.zip (or __module__.zip, or whatever)

So you can have only a single multi-file non-package zipfile in each
directory on sys.path? I find this quite ugly: a directory with only a
single file in it, and potentially many of these. What is the
advantage of this limitation?

Regards,
Martin



From pedronis@bluewin.ch  Thu Dec  5 23:31:45 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Fri, 6 Dec 2002 00:31:45 +0100
Subject: [Python-Dev] Another approach for the import mechanism
References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico> <20021205212127.G19843@ibook.distro.conectiva>
Message-ID: <0d8401c29cb6$79c48b60$6d94fea9@newmexico>

>
> If I want to provide packaged RPMs for small systems, I'd like
> that *every* module/package which can be installed optionally, be
> packaged inside a compressed file. There are real usage for
> that right now.
>

Not that I exactly understand what you want, but it seems that some logic in
site.py, such that all *.zip found e.g. in site-archives (and similars)
(thought as equivalent and parallel to site-packages) are added to sys.path,
would solve your problem.

You either install a dir hiearchy somewhere or an archive there.

I insist that supporting compressing single modules seems YAGNI and requiring
too much complexity.

regards.



From pedronis@bluewin.ch  Thu Dec  5 23:52:21 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Fri, 6 Dec 2002 00:52:21 +0100
Subject: [Python-Dev] Another approach for the import mechanism
References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva>
Message-ID: <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico>

From: "Gustavo Niemeyer" <niemeyer@conectiva.com>
> - Allows me to ship a package inside a zip file, without asking
>   the user to change his path.
>

btw for single packages (once you can put zipfiles in sys.path or __path__) you
can achieve this
with __path__

package/
  __init__.py

    __path__[0] = os.path.join(__path__[0],'package.zip')

  package.zip

regards.



From brian@sweetapp.com  Fri Dec  6 00:14:53 2002
From: brian@sweetapp.com (Brian Quinlan)
Date: Thu, 05 Dec 2002 16:14:53 -0800
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <m3vg2859d6.fsf@mira.informatik.hu-berlin.de>
Message-ID: <00a901c29cbc$803d2230$21795418@dell1700>

> Of course, if somebody wants to provide a crypt importer (which
> decrypts the source before importing it) in addition to the
> pybz2importer, then you get the same coordination problem as with all
> prior import hooks.

Does anybody really believe that this problem can be solved in general?

I like the following:

1. only strings representing directories are allowed in sys.path 
  (a zip archive is conceptually a directory)
2. there are two types of hook that can be installed:
     - an import hook that returns a module object, a stream object
       or None
     - a stream hook that has the opportunity to return a different
       stream if it likes

When the import hook returns a stream, every stream hook is given a
chance to transform that stream.  If a stream hook returns a transformed
stream then every other hook is called again. This continues until no
stream hook is interested in transforming the stream anymore. 

I know that this might have performance issues, but it seems like it
would allow you to have compressed archives of encrypted modules located
on a web server without any of the hooks knowing anything about each
other.

Cheers,
Brian



From bac@OCF.Berkeley.EDU  Fri Dec  6 00:24:52 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Thu, 5 Dec 2002 16:24:52 -0800 (PST)
Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30
In-Reply-To: <20021205173053.A18494@ibook.distro.conectiva>
References: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU>
 <20021205173053.A18494@ibook.distro.conectiva>
Message-ID: <Pine.SOL.4.50.0212051620180.28199-100000@death.OCF.Berkeley.EDU>

[Gustavo Niemeyer]

> > ================================
> > `Dictionary evaluation order`__
> > ================================
> > __ http://mail.python.org/pipermail/python-dev/2002-November/030458.html
> >
> [...]
> > Finn Bock said that Jython went f1, f2, f3, f4.  In that case Guido didn't
> > see any reason to fix it.  But Tim Peters brought up the point that the
> [...]
>
> I belive the correct sentence here would be:
>
> "In that case Guido didn't see any reason to *block* the fix."
>

Oops.  =)

So this brings up something I have been pondering; what to do when a
mistake in a summary is found after it is already out the door.  I can
obviously fix the web version, but is it really worth it since I think
most people read the text version that goes to
comp.lang.python(.announce)? .  Does anyone object if I just leave it
as-is unless it is a major error that changes the meaning of the summary?
This obviously makes it seems like Guido said something he didn't, but the
outcome in the end is the same; the change was made.

And I don't want to have to go back and make changes to summaries that are
6 months old or something when the issue has long past.

So, keep the errors in the archive or change them?

-Brett


From paul@pfdubois.com  Fri Dec  6 01:02:26 2002
From: paul@pfdubois.com (Paul F Dubois)
Date: Thu, 5 Dec 2002 17:02:26 -0800
Subject: [Python-Dev] PEP 242 Numeric kinds
Message-ID: <000001c29cc3$281ed420$6501a8c0@NICKLEBY>

We had put off consideration of PEP 242 Numeric kinds until June 2002
when a meeting of some interested parties was to occur but the meeting
didn't occur. I have a draft implementation and users of it but my
feeling is that although correct and useful the PEP is not useful enough
to put it in the standard library. Since it really only interests
scientific programmers I propose simply making it a separate deliverable
on the Numeric site and withdrawing the PEP.



From skip@pobox.com  Fri Dec  6 01:22:35 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 5 Dec 2002 19:22:35 -0600
Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30
In-Reply-To: <Pine.SOL.4.50.0212051620180.28199-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU>
 <20021205173053.A18494@ibook.distro.conectiva>
 <Pine.SOL.4.50.0212051620180.28199-100000@death.OCF.Berkeley.EDU>
Message-ID: <15855.64475.165789.875723@montanaro.dyndns.org>

    Brett> So this brings up something I have been pondering; what to do
    Brett> when a mistake in a summary is found after it is already out the
    Brett> door.

Fix what you can. (tm)

Skip


From niemeyer@conectiva.com  Fri Dec  6 02:11:02 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Fri, 6 Dec 2002 00:11:02 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico>
References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico>
Message-ID: <20021206001101.A21140@ibook.distro.conectiva>

> btw for single packages (once you can put zipfiles in sys.path or
> __path__) you can achieve this with __path__
> 
> package/
>   __init__.py
>     __path__[0] = os.path.join(__path__[0],'package.zip')
> 
>   package.zip

That would kill the __init__.py that could be inside package.zip,
right? One could leave package.zip's __init__.py outside it, and
hack it as shown above, but it'd be great if this scheme was just
an option for setup.py (--compress-packages). Otherwise the scheme
must be prepared by the developer, not by the packager.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From niemeyer@conectiva.com  Fri Dec  6 02:17:36 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Fri, 6 Dec 2002 00:17:36 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <m3vg2859d6.fsf@mira.informatik.hu-berlin.de>
References: <20021205192616.B18494@ibook.distro.conectiva> <005701c29ca9$d8517ba0$ced241d5@hagrid> <20021205203137.C19843@ibook.distro.conectiva> <09e901c29cb0$0da094c0$6d94fea9@newmexico> <20021205212127.G19843@ibook.distro.conectiva> <m3vg2859d6.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021206001736.B21140@ibook.distro.conectiva>

> > Yes, it is addressed. My propose is to allow the importer mechanism to
> > check if there's something importable with the given name in the given
> > path, instead of telling him what to import. With that in mind,
> > importing foobar.py.bz2 is just a matter of checking for
> > <modulename>.py.bz2 in the given path.
> 
> It appears that this can't be really done with the proposed mechanism,
> as you want to have a multiple hooks for a single source "URL".

As you have mentioned below, all hooks registered are called, if
the previous ones failed. So there shouldn't be any problem.

> If you *only* want to have a .py.bz2 importer, you could replace every
> directory on sys.path with the pybz2importer, which would check for
> .py.bz2 files in the directory, and, if that fails, calls
> imp.find_module.
> 
> Of course, if somebody wants to provide a crypt importer (which
> decrypts the source before importing it) in addition to the
> pybz2importer, then you get the same coordination problem as with all
> prior import hooks.

Yes, that's the idea behind all proposes after all.

> So you are back to calling all registered hooks for all items on
> sys.path, which might be expensive.

Agreed. But further hooks are only called if the default one failed.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From j_r_fonseca@yahoo.co.uk  Fri Dec  6 02:37:28 2002
From: j_r_fonseca@yahoo.co.uk (=?iso-8859-15?Q?Jos=E9?= Fonseca)
Date: Fri, 6 Dec 2002 02:37:28 +0000
Subject: [Python-Dev] Re: [Numpy-discussion] PEP 242 Numeric kinds
In-Reply-To: <000001c29cc3$281ed420$6501a8c0@NICKLEBY>
References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY>
Message-ID: <20021206023728.GA24182@localhost.localdomain>

On Thu, Dec 05, 2002 at 05:02:26PM -0800, Paul F Dubois wrote:
> Since it really only interests
> scientific programmers I propose simply making it a separate deliverable
> on the Numeric site and withdrawing the PEP.

I'm not fully aware about the advantages/implications of puting Numeric
in Python standard library, but to say that the manipulation of numeric
arrays is only of interest to scientific programmers is the same of when
in the early computing days engineers saing that computers would only be
good for crunching numbers, and that the concept of personal computers
was just non-sense...

For a non-scientic usage of Numeric see the examples in
http://www.pygame.org/pcr/repository.php, but I can imagine the
usefullness of Numeric in many more non-scientific applications:
imaging, sound visualization plugins, 3D graphics, and probably much
more.

The use of Numeric can speed up alot of algorithms which would be
otherwise very slow in pure Python, and therefore forcing one to write C
extensions. That's why IMHO something with the Numeric functionality
should exist in the Python standard library.

Note that this is not the same that saying that Numeric should be
included as is - perhaps it's better to have it seperately to let it
mature more, perhaps not - but still, there is much more than a niche
interest around it.

José Fonseca
__________________________________________________
Do You Yahoo!?
Everything you'll ever need on one web page
from News and Sport to Email and Music Charts
http://uk.my.yahoo.com


From niemeyer@conectiva.com  Fri Dec  6 02:53:52 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Fri, 6 Dec 2002 00:53:52 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <m3r8cw5964.fsf@mira.informatik.hu-berlin.de>
References: <20021205192616.B18494@ibook.distro.conectiva> <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de> <20021205203312.A20315@ibook.distro.conectiva> <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de> <20021205212604.H19843@ibook.distro.conectiva> <m3r8cw5964.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021206005352.C21140@ibook.distro.conectiva>

> So you can have only a single multi-file non-package zipfile in each
> directory on sys.path? I find this quite ugly: a directory with only a

Well, with the current implementation you'll only allow one single
multi-file non-package zipfile in each entry of sys.path. :-)

> single file in it, and potentially many of these. What is the
> advantage of this limitation?

- Don't have to change path to use compressed packages (at least
  not if you want to provide compressed packages, individual
  compressed modules or the standard library).

- Don't have to specify the compression type hardcoded.

- Allows one to ship a package inside a zip file, without asking
  the user to change his path, and without hacking the package.

- Allows one to compress a single file (foobar.py.bz2).

I belive that my propose is quite clear now. If there are no
additional supporters, there's no reason to go on.

Thanks to everyone who discussed.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From niemeyer@conectiva.com  Fri Dec  6 03:05:45 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Fri, 6 Dec 2002 01:05:45 -0200
Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30
In-Reply-To: <Pine.SOL.4.50.0212051620180.28199-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU> <20021205173053.A18494@ibook.distro.conectiva> <Pine.SOL.4.50.0212051620180.28199-100000@death.OCF.Berkeley.EDU>
Message-ID: <20021206010545.D21140@ibook.distro.conectiva>

[...]
> Does anyone object if I just leave it as-is unless it is a major error
> that changes the meaning of the summary?  This obviously makes it

This specific case changes the meaning of the summary. It puts
the BDFL in the exact oposite side, turning me into a subversor. ;-)

[...]
> So, keep the errors in the archive or change them?

Well.. I don't really care about that specific case. But IMHO, if it's
easy to fix, information should be changed at least in the web, even 6
months later, since there's no point in providing misgiven information
if we know it's wrong.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From greg@cosc.canterbury.ac.nz  Fri Dec  6 03:09:24 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Fri, 06 Dec 2002 16:09:24 +1300 (NZDT)
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <00a901c29cbc$803d2230$21795418@dell1700>
Message-ID: <200212060309.gB639OP12627@kuku.cosc.canterbury.ac.nz>

Brian Quinlan <brian@sweetapp.com>:

> When the import hook returns a stream, every stream hook is given a
> chance to transform that stream.  If a stream hook returns a transformed
> stream then every other hook is called again. This continues until no
> stream hook is interested in transforming the stream anymore. 

How is a stream hook to know what kind of stream it's
been given?

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From brian@sweetapp.com  Fri Dec  6 03:37:19 2002
From: brian@sweetapp.com (Brian Quinlan)
Date: Thu, 05 Dec 2002 19:37:19 -0800
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <200212060309.gB639OP12627@kuku.cosc.canterbury.ac.nz>
Message-ID: <00b101c29cd8$c7b25f60$21795418@dell1700>

Greg Ewing wrote:
> > When the import hook returns a stream, every stream hook is given a
> > chance to transform that stream.  If a stream hook returns a
transformed
> > stream then every other hook is called again. This continues until
no
> > stream hook is interested in transforming the stream anymore.
> 
> How is a stream hook to know what kind of stream it's
> been given?

It will have to figure it out itself, given the original arguments to
the import hook plus the stream itself.

A zip hook might, for example, check that the first 4 bytes of the
stream are 0x04034b50. If they aren't, it would have to repair the
original stream (haven't really thought about how to do this) and signal
its lack of interest. Otherwise it would open the archive, decompress it
and return a new stream. 

The more I think about this mechanism, the fewer interesting problems it
seems to solve.

Maybe the real answer is to have a single import hook but write as much
of it as possible in modular Python. Then the user can figure out how to
manage the competing interests when importing archived, encrypted
modules from an arbitrary URL.

Cheers,
Brian



From aahz@pythoncraft.com  Fri Dec  6 04:02:58 2002
From: aahz@pythoncraft.com (Aahz)
Date: Thu, 5 Dec 2002 23:02:58 -0500
Subject: [Python-Dev] Import: where's the PEP?
Message-ID: <20021206040258.GA24214@panix.com>

Having just slogged my way through the entire import thread so far, I've
got to ask why nobody else has demanded a PEP before we go any further?
There are too many competing proposals; we need a good summary of what
design we're planning, plus a summary of why all other proposals are
rejected.

I'm a very strong -1 on anything until there's a PEP.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From skip@pobox.com  Fri Dec  6 04:33:59 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 5 Dec 2002 22:33:59 -0600
Subject: [Python-Dev] Import: where's the PEP?
In-Reply-To: <20021206040258.GA24214@panix.com>
References: <20021206040258.GA24214@panix.com>
Message-ID: <15856.10423.787013.625507@montanaro.dyndns.org>

    aahz> I'm a very strong -1 on anything until there's a PEP.

Try PEP 273.  I don't believe Just's original proposal meant to go beyond
that.

Skip


From aahz@pythoncraft.com  Fri Dec  6 04:39:21 2002
From: aahz@pythoncraft.com (Aahz)
Date: Thu, 5 Dec 2002 23:39:21 -0500
Subject: [Python-Dev] Import: where's the PEP?
In-Reply-To: <15856.10423.787013.625507@montanaro.dyndns.org>
References: <20021206040258.GA24214@panix.com> <15856.10423.787013.625507@montanaro.dyndns.org>
Message-ID: <20021206043921.GA10641@panix.com>

On Thu, Dec 05, 2002, Skip Montanaro wrote:
> 
>     aahz> I'm a very strong -1 on anything until there's a PEP.
> 
> Try PEP 273.  I don't believe Just's original proposal meant to go beyond
> that.

Doesn't look that way to me: Just is, after all, proposing a brand-new
import hook mechanism.  His proposal is *compatible* with PEP 273, but
it's definitely not the same thing.  That's not even talking about all
the bandwidth that's been consumed with other ideas -- the whole point
of the PEP mechanism is to avoid losing that valuable information!
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From python@rcn.com  Fri Dec  6 05:32:03 2002
From: python@rcn.com (Raymond Hettinger)
Date: Fri, 6 Dec 2002 00:32:03 -0500
Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30
References: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU> <20021205173053.A18494@ibook.distro.conectiva> <Pine.SOL.4.50.0212051620180.28199-100000@death.OCF.Berkeley.EDU> <20021206010545.D21140@ibook.distro.conectiva>
Message-ID: <002101c29ce8$d1ba7a00$770ca044@oemcomputer>

> > Does anyone object if I just leave it as-is unless it is a major error
> > that changes the meaning of the summary?  

Ideally, it would be best to fix *anything* that is incorrect.


<snip>
> > So, keep the errors in the archive or change them?

As a recorder of history, accuracy takes precedence.
In the archive, keep the original, add a bracketed section
entitled Subsequent Corrections or some such, and make
it right.

In the next summary, add an errata section, and post the
correction.

Errata and amendments are like bugfixes, everyone admires
the effort and knows they are better off with honest attempts
at accuracy than they are with someone asserting that they
always had it right to begin with.  It's just good journalism

Thanks again for your efforts to keep the community informed.



Raymond


From martin@v.loewis.de  Fri Dec  6 06:25:37 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 07:25:37 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021206005352.C21140@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de>
 <20021205203312.A20315@ibook.distro.conectiva>
 <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de>
 <20021205212604.H19843@ibook.distro.conectiva>
 <m3r8cw5964.fsf@mira.informatik.hu-berlin.de>
 <20021206005352.C21140@ibook.distro.conectiva>
Message-ID: <m3fztbfyu6.fsf@mira.informatik.hu-berlin.de>

Gustavo Niemeyer <niemeyer@conectiva.com> writes:

> > So you can have only a single multi-file non-package zipfile in each
> > directory on sys.path? I find this quite ugly: a directory with only a
> 
> Well, with the current implementation you'll only allow one single
> multi-file non-package zipfile in each entry of sys.path. :-)

Yes, but I don't have to clutter my disk for that.

> - Don't have to change path to use compressed packages (at least
>   not if you want to provide compressed packages, individual
>   compressed modules or the standard library).

I thought you just explained that I will need to change the path to
provide a compressed standard library, to point to a directory that
contains an __init__.zip.

> - Don't have to specify the compression type hardcoded.

I don't understand that remark. If I have a zipfile, I surely must
install a zipfile hook in your approach also - a .tar.bz2 hook won't
be able to load the zipfile, no?

> - Allows one to ship a package inside a zip file, without asking
>   the user to change his path, and without hacking the package.
> 
> - Allows one to compress a single file (foobar.py.bz2).

This is really the same issue: If you had a mechanism to import a
module from a .py.bz2 file, you could use the same mechanism to import
a package (or subpackage) from a .zip file. While I think this might
be desirable, I also think it was never the goal of PEP 273 to provide
such a facility.

Regards,
Martin


From martin@v.loewis.de  Fri Dec  6 06:28:12 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 07:28:12 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021206001101.A21140@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <005701c29ca9$d8517ba0$ced241d5@hagrid>
 <20021205203137.C19843@ibook.distro.conectiva>
 <0ef401c29cb9$5a8a6fa0$6d94fea9@newmexico>
 <20021206001101.A21140@ibook.distro.conectiva>
Message-ID: <m3bs3zfypv.fsf@mira.informatik.hu-berlin.de>

Gustavo Niemeyer <niemeyer@conectiva.com> writes:

> That would kill the __init__.py that could be inside package.zip,
> right? One could leave package.zip's __init__.py outside it, and
> hack it as shown above, but it'd be great if this scheme was just
> an option for setup.py (--compress-packages). Otherwise the scheme
> must be prepared by the developer, not by the packager.

You can easily put an entire package into a .zip file, as the
packager: Just provide a .zip file with the entire package contents
(and file names starting with the package dir inside the zip file);
then provide a .pth file to add the zipfile to sys.path.

Regards,
Martin



From martin@v.loewis.de  Fri Dec  6 06:34:16 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 07:34:16 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <20021206001736.B21140@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva>
 <005701c29ca9$d8517ba0$ced241d5@hagrid>
 <20021205203137.C19843@ibook.distro.conectiva>
 <09e901c29cb0$0da094c0$6d94fea9@newmexico>
 <20021205212127.G19843@ibook.distro.conectiva>
 <m3vg2859d6.fsf@mira.informatik.hu-berlin.de>
 <20021206001736.B21140@ibook.distro.conectiva>
Message-ID: <m37kenfyfr.fsf@mira.informatik.hu-berlin.de>

Gustavo Niemeyer <niemeyer@conectiva.com> writes:

> > It appears that this can't be really done with the proposed mechanism,
> > as you want to have a multiple hooks for a single source "URL".
> 
> As you have mentioned below, all hooks registered are called, if
> the previous ones failed. So there shouldn't be any problem.

I mentioned this as a theoretical architecture which is not going to
be implemented by Just. Whether or not this is a problem, I don't
know.

> > So you are back to calling all registered hooks for all items on
> > sys.path, which might be expensive.
> 
> Agreed. But further hooks are only called if the default one failed.

Again, this is *not* the strategy that Just proposes, and, unless
somebody provides an implementation of it, not one that will matter.

As for "only called if the default one failed": This will be the
normal case. In my standard installation, sys.path is

['', '/home/martin/work', '/usr/local/lib/python2.3',
'/usr/local/lib/python2.3/plat-linux2',
'/usr/local/lib/python2.3/lib-tk',
'/usr/local/lib/python2.3/lib-dynload',
'/usr/local/lib/python2.3/site-packages',
'/usr/local/lib/site-python']

So if I have 5 import hooks (including the default one), and I do
"import Fnorb" (which is in site-python), I get 36 hook calls before
the Fnorb package is found. For each element in sys.path, first the
default hook will fail, and then all additional hooks, then it
proceeds to the next element of sys.path. Quite expensive,
potentially.

Regards,
Martin



From martin@v.loewis.de  Fri Dec  6 06:40:47 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 07:40:47 +0100
Subject: [Python-Dev] Import: where's the PEP?
In-Reply-To: <20021206043921.GA10641@panix.com>
References: <20021206040258.GA24214@panix.com>
 <15856.10423.787013.625507@montanaro.dyndns.org>
 <20021206043921.GA10641@panix.com>
Message-ID: <m33cpbfy4w.fsf@mira.informatik.hu-berlin.de>

Aahz <aahz@pythoncraft.com> writes:

> Doesn't look that way to me: Just is, after all, proposing a brand-new
> import hook mechanism.  His proposal is *compatible* with PEP 273, but
> it's definitely not the same thing.  That's not even talking about all
> the bandwidth that's been consumed with other ideas -- the whole point
> of the PEP mechanism is to avoid losing that valuable information!

I agree that a PEP should be written for anything that is going to be
accepted; I disagree that the whole point of the PEP mechanism is to
avoid losing valuable information.

Instead, I think the purpose of the PEP processis to focus the
discussion, so all people know they are talking about the same thing,
and you can better tell what remarks are off-topic, and avoid endless
repetition of the same observations.

In this discussion, I think PEP 273 helped to focus the discussion, as
every proposal for a new import mechanism had to show that it can
provide the feature of that PEP (i.e. zipfile imports).

Regards,
Martin



From python@rcn.com  Fri Dec  6 06:56:25 2002
From: python@rcn.com (Raymond Hettinger)
Date: Fri, 6 Dec 2002 01:56:25 -0500
Subject: [Python-Dev] PEP 242 Numeric kinds
References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY>
Message-ID: <002701c29cf4$98296c40$770ca044@oemcomputer>

> I have a draft implementation and users of it but my
> feeling is that although correct and useful the PEP is not useful enough
> to put it in the standard library. Since it really only interests
> scientific programmers I propose simply making it a separate deliverable
> on the Numeric site and withdrawing the PEP.

As the PEP author, your feeling is probably the most accurate.
And, it is true that Kinds will, in general, appeal to the same crowd 
as Numeric.

OTOH, "import kinds" is entirely unobtrusive and general purpose.
The interface is simple; and may help some code become less platform
sensitive (in the way the Metafont writes its own portable routines to 
create consistent lettering); and it has some educational value (making
it easy to demonstrate the effects of truncation and floating point
round-off).  

Also, I think accounting applications can benefit from having some
positive assurance that multi-billion dollar calculations don't lose 
their pennies and that the books will still balance.

So, the only downside is having an additional block of code to 
maintain.  My vote is for adding it to the standard library.  Control
over precision is more of a core capability than it is an extension,
but, as the PEP author and principal user, you probably know best.

'nuff said,


Raymond Hettinger


#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################


From bac@OCF.Berkeley.EDU  Fri Dec  6 07:12:44 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Thu, 5 Dec 2002 23:12:44 -0800 (PST)
Subject: [Python-Dev] PEP 242 Numeric kinds
In-Reply-To: <000001c29cc3$281ed420$6501a8c0@NICKLEBY>
References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY>
Message-ID: <Pine.SOL.4.50.0212052304510.28199-100000@death.OCF.Berkeley.EDU>

[Paul F Dubois]

> We had put off consideration of PEP 242 Numeric kinds until June 2002
> when a meeting of some interested parties was to occur but the meeting
> didn't occur. I have a draft implementation and users of it but my
> feeling is that although correct and useful the PEP is not useful enough
> to put it in the standard library. Since it really only interests
> scientific programmers I propose simply making it a separate deliverable
> on the Numeric site and withdrawing the PEP.
>

I say add it.  If there are already users out there then that demonstrates
as least some form of a demand.

Besides, if rationals can get into the library (that module was finally
accepted to be added to the library, right?) then a module to help do
consistent decimal math should at least be included at least until the
Python core can move to C99 (if I am remembering a comment from Tim saying
that C99 adds more explicit numeric types, e.g., exactly 2 byte float).

I just want to be able to add up a bunch of money values and not have a
few pennies disappear in the end; usually I am the one who has to eat the
loss and I am an poor college graduate who can't afford the rounding.  =)

-Brett


From fredrik@pythonware.com  Fri Dec  6 08:20:32 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 09:20:32 +0100
Subject: [Python-Dev] Another approach for the import mechanism
References: <20021205192616.B18494@ibook.distro.conectiva><005701c29ca9$d8517ba0$ced241d5@hagrid><20021205203137.C19843@ibook.distro.conectiva><09e901c29cb0$0da094c0$6d94fea9@newmexico><20021205212127.G19843@ibook.distro.conectiva><m3vg2859d6.fsf@mira.informatik.hu-berlin.de><20021206001736.B21140@ibook.distro.conectiva> <m37kenfyfr.fsf@mira.informatik.hu-berlin.de>
Message-ID: <006601c29d00$59936d30$0900a8c0@spiff>

martin wrote:

> Again, this is *not* the strategy that Just proposes, and, unless
> somebody provides an implementation of it, not one that will matter.

Depends on how you measure success, of course.  Working code that
addresses problems observed in the wild, or the length of the discussion
thread on a mailing list.

</F>



From Paul.Moore@atosorigin.com  Fri Dec  6 09:38:50 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 6 Dec 2002 09:38:50 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E67@UKDCX001.uk.int.atosorigin.com>

From: Gordon McMillan [mailto:gmcm@hypernet.com]
> BTW, iu does allow non string objects in sys.path.
> If it exposes "getmod(name)", that will be tried
> in preference to iu's own machinery.

I just tried this out. As far as I can see, iu works perfectly
with a simple getmod() method. Why does your patch require
find_module() and load_module()?

I'm not criticising, just trying to understand the
requirements... (Anything to make writing an import hook
easier!!!)

You'll notice I'm wavering over my dislike of having objects on
sys.path :-)

My interest in import hooks stems from wanting to be able to
implement something like Java's 'executable jar file' support,
where an application, data and all, can be packaged up as a
single file. In theory, there's no real problem with this (I can
write import hooks to load from a zip file right now), but in
practice, the bootstrapping problems are quite nasty. The more
in-core (or at least standard library) support there is for these
features, the less of a bootstrap problem I have...

Paul.

"""
Example code for iu (adds an object to sys.path which satisfies *all*
requests for modules and packages with a new empty module...)
"""

import iu
import imp
import sys

iu.ImportManager().install()

class Tester:
    def getmod(self, name):
        print "Requested", name, locals()
        m =3D imp.new_module(name)
        # Either set __path__ to contain an object,
        # or set __path__ to contain a path, and
        # __importsub__ to the object...
        m.__path__ =3D [ Tester() ]
        sys.modules[name] =3D m
        return m

sys.path.append(Tester())


From fredrik@pythonware.com  Fri Dec  6 09:54:18 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 10:54:18 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <16E1010E4581B049ABC51D4975CEDB885E2E67@UKDCX001.uk.int.atosorigin.com>
Message-ID: <018001c29d0d$72a50bf0$0900a8c0@spiff>

Paul Moore wrote:
>=20
> I just tried this out. As far as I can see, iu works perfectly
> with a simple getmod() method. Why does your patch require
> find_module() and load_module()?
>=20
> I'm not criticising, just trying to understand the
> requirements... (Anything to make writing an import hook
> easier!!!)

the internal import architecture does things in two steps, for
some reason.  this is also reflected in the public "imp" module
interface:

    http://www.python.org/doc/2.2.2/lib/module-imp.html

I'm not sure how Just's patch interacts with "imp", so maybe
this is a non-issue. =20

</F>



From Paul.Moore@atosorigin.com  Fri Dec  6 10:13:24 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 6 Dec 2002 10:13:24 -0000
Subject: [Python-Dev] Another approach for the import mechanism
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619976@UKDCX001.uk.int.atosorigin.com>

Samuele Pedroni wrote:
> btw for single packages (once you can put zipfiles in sys.path
> or __path__) you can achieve this with __path__
>
> package/
>   __init__.py
>
>     __path__[0] =3D os.path.join(__path__[0],'package.zip')
>
>   package.zip

Of course, if __path__[0] was originally an object, this would
break...

I think that the objects-on-sys.path issue has now been resolved,
but this probably acts as a reasonable "real world" example of what
needs to be supported. (Unless I've misunderstood things again :-))

Paul.


From just@letterror.com  Fri Dec  6 10:13:10 2002
From: just@letterror.com (Just van Rossum)
Date: Fri,  6 Dec 2002 11:13:10 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <018001c29d0d$72a50bf0$0900a8c0@spiff>
Message-ID: <r01050400-1022-78DEC7E8090311D7AD06003065D5E7E4@[10.0.0.23]>

Fredrik Lundh wrote:

> the internal import architecture does things in two steps, for
> some reason.  this is also reflected in the public "imp" module
> interface:
> 
>     http://www.python.org/doc/2.2.2/lib/module-imp.html
> 
> I'm not sure how Just's patch interacts with "imp", so maybe
> this is a non-issue.  

The patch doesn't yet, but that's only to keep the patch small for initial
review purposes. I have some thoughts about how it _could_ interact with imp,
but that needs more thought. The whole find/load separation is still in tact,
and is reflected in the zipimport.zipimporter object: it has a find_module() and
a load_module() method. I'm not sure if the two-step approach is a big enough
deal to keep it in a new import scheme.

Just


From fredrik@pythonware.com  Fri Dec  6 10:40:50 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 11:40:50 +0100
Subject: [Python-Dev] New and Improved Import Hooks
References: <r01050400-1022-78DEC7E8090311D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <025801c29d13$f30f97a0$0900a8c0@spiff>

just wrote:

> I'm not sure if the two-step approach is a big enough deal to
> keep it in a new import scheme.

does anyone remember/know why it was done in two steps?

</F>



From martin@v.loewis.de  Fri Dec  6 10:47:46 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 11:47:46 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619976@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619976@UKDCX001.uk.int.atosorigin.com>
Message-ID: <m3n0njjuel.fsf@mira.informatik.hu-berlin.de>

"Moore, Paul" <Paul.Moore@atosorigin.com> writes:

> >     __path__[0] = os.path.join(__path__[0],'package.zip')
> >
> >   package.zip
> 
> Of course, if __path__[0] was originally an object, this would
> break...

What do you mean with "was originally", and what do you mean with
"would break"?

__path__[0] cannot be an object originally, because the package loader
inserts a string, always.

Even if it was a (non-string) object, I can't see how this would
break. os.path.join would continue to work if the object is a string
subclass, or atleast supports len, __getitem__, and __add__.

The result would be a string, which would be converted to a zipimport
on the next package contents import.

> I think that the objects-on-sys.path issue has now been resolved,
> but this probably acts as a reasonable "real world" example of what
> needs to be supported. (Unless I've misunderstood things again :-))

I recommend you try the patch, to see how it works in the real
world.

Regards,
Martin


From Paul.Moore@atosorigin.com  Fri Dec  6 10:56:02 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 6 Dec 2002 10:56:02 -0000
Subject: [Python-Dev] Another approach for the import mechanism
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com>

From: Martin v. L=F6wis [mailto:martin@v.loewis.de]
> What do you mean with "was originally", and what do you mean
> with "would break"?
[...]
Looks like I did misunderstand :-(

> I recommend you try the patch, to see how it works in the real
> world.

At the moment the patch crashes immediately on Windows. I've let
Just know, and I'll try the next version when it arrives...

Paul.

PS If Just's patch *doesn't* stabilise in time for 2.3, and we
   go back to Jim Ahlstrom's patch, I'd support Just writing a
   PEP to cover his import hooks mechanism, for future inclusion.
   (Actually, I think that Just's mechanism has sufficient extra
   functionality over PEP 273, that it deserves a PEP even now.
   But if Just takes time out to write a PEP, I suspect that
   makes it even less likely we'll get working code in time...)


From just@letterror.com  Fri Dec  6 11:01:23 2002
From: just@letterror.com (Just van Rossum)
Date: Fri,  6 Dec 2002 12:01:23 +0100
Subject: [Python-Dev] zipimport & import hooks
Message-ID: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]>

I think we can separate the zipimport issue from the import hooks issue, but
only to an extent. *Some* hook need to be installed in import.c for zipimports
to work, but it doesn't neccesarily mean we must add a general new import hook
scheme *now*. But then again, it might be a *very* small step from enabling
zipimports to exposing the needed hook and have a clean and general solution.

It seems the majority doesn't like non-strings on sys.path, which to me is
understandable, especially if you take PYTHONPATH into consideration. Let's try
to focus on that.

Here's a minimal implementation idea that will allow .zip filenames on sys.path:

    zipimporters = {}  # a cache, private to import.c
    for p in sys.path:
        if p.endswith(".zip"):
            z = zipimporters.get(p)
            if z is None:
                z = zipimporters[p] = zipimporter(p)
            loader = z.find_module(name)
            ...etc...
        else:
            ...builtin import...

This gets nessesarily more complex for packages, as now we also have to cater
for __path__ items of this form: "/path/to/my/archive.zip/packagedir". In this
case we need to create a "subimporter" of "/path/to/my/archive.zip" if only
because we don't want to reread the zip archive's file index. By now the
machinery needed can be written like so:

    zipimporters = {}  # a cache, private to import.c

    def get_zipimporter(p):
        if p.endswith(".zip"):
            return zipimporter(p)
        if not os.path.exists(p):
            pos = p.rfind(".zip")
            if pos > 0:
                archive = p[:pos + 4]
                subpath = p[pos + 5:]  # skip initial sep
                z = zipimporters[archive]
                return z.subimporter(subpath)

    for p in sys.path:
        importer = zipimporters.get(p, -1)
        if importer != -1:
            importer = zipimporters[p] = get_importer(p)
        if importer is not None:
            loader = importer.find_module(name)
            ...etc...
        else:
            ...builtin import...

And *this* can easily be rewritten in a slightly more general form:

    path_importers = {}  # exposed in sys
    import_hooks = [get_zimpimporter]  # exposed in sys

    # get_zimpimporter() implementation left out for brevity

    def get_importer(p):
        for hook in import_hooks:
            importer = hook(path)
            if importer is not None:
                return None
        return None

    for p in sys.path:
        importer = path_importers.get(p, -1)
        if importer != -1:
            importer = path_importers[p] = get_importer(p)
        if importer is not None:
            loader = importer.find_module(name)
            ...etc...
        else:
            ...builtin import...
            
Which, an observant reader might notice, is a stripped down version of Gordon's
iu.py, with s/shadowpath/path_importers/ and s/ownertypes/import_hooks/ and
without his metapath. (The metapath is a great idea, but I don't think it's
possible to do without a *major* rewrite of import.c. The rest is relatively
easy to do, in fact I've got it mostly working already.)

Just


From Paul.Moore@atosorigin.com  Fri Dec  6 11:17:36 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 6 Dec 2002 11:17:36 -0000
Subject: [Python-Dev] Another approach for the import mechanism
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com>

From: Martin v. L=F6wis [mailto:martin@v.loewis.de]
> __path__[0] cannot be an object originally, because the
> package loader inserts a string, always.

Thinking some more, I'd dispute that "always". The *current*
package loader inserts a string, yes. Under Just's patch,
I think this remains true, as well.

But for a general import hook, it certainly isn't enforced. I
posted a snippet of code which uses Gordon's iu, which works
perfectly but which inserts an object into a package's __path__.
I'm not saying that this isn't bad practice, or even relevant to
the case being discussed, but it's *certainly* true that this
whole area is under-documented and not properly enforced. The
only reference I could find to package structure in the standard
documentation was "[XXX Can't be bothered to spell this out right
now; see the URL http://www.python.org/doc/essays/packages.html
for more details, also about how the module search works from
inside a package.]". That URL points to a document which is
described as "Built-in Package Support in Python 1.5" which
doesn't inspire confidence in its accuracy for current Python...
[Hey, I just noticed that it documents __all__, which wasn't in
Python 1.5, I believe. So maybe it is up to date after all!]

Actually, there's a piece of sample code in that document which
suggests putting platform-specific code into the search path via

    dirname =3D __path__[0]		# Package's main folder
    __path__.insert(0, os.path.join(dirname, "plat-" + platform))

That would result in things like ".../lib/package.zip/plat-win32"
being on __path__. Will Just's code handle this? (I'd check, but
the code doesn't run on Win32 yet. Looking at the code seems to
indicate that it will not handle this...) FWIW, Jim's patch does.

Paul.


From martin@v.loewis.de  Fri Dec  6 11:18:10 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 12:18:10 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com>
Message-ID: <m3adjjjszx.fsf@mira.informatik.hu-berlin.de>

"Moore, Paul" <Paul.Moore@atosorigin.com> writes:

>    (Actually, I think that Just's mechanism has sufficient extra
>    functionality over PEP 273, that it deserves a PEP even now.
>    But if Just takes time out to write a PEP, I suspect that
>    makes it even less likely we'll get working code in time...)

Indeed. It appears that, if Just completes the patch, we need BDFL
pronouncement, anyway, to chose one of the following options:

- Apply Jim's patch: Ugly, unreadable code, but provides strictly
  PEP 273 only, and preserves backwards compatibility.

- Apply Just's patch: Nice code, probably provides PEP 273, might
  break compatibility, does not provide all features it claims to
  provide (you can't implement a .py.bz2 importer with it).

- Wait for YAGNI (yet another great new infrastructure), to invoke all
  hooks on all sys.path items; this would give backwards compatibility
  (sys.path could continue to be a list of strings), extensibility
  (you could add new strings to sys.path, yet the hooks would honor
  them), feature-richness (you can implement a .py.bz2 importer),
  and slowness (for invoking every hook on every sys.path item).

Regards,
Martin



From Paul.Moore@atosorigin.com  Fri Dec  6 11:26:28 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 6 Dec 2002 11:26:28 -0000
Subject: [Python-Dev] Another approach for the import mechanism
Message-ID: <16E1010E4581B049ABC51D4975CEDB885E2E6B@UKDCX001.uk.int.atosorigin.com>

From: Martin v. L=F6wis [mailto:martin@v.loewis.de]

> "Moore, Paul" <Paul.Moore@atosorigin.com> writes:
>=20
> >    (Actually, I think that Just's mechanism has sufficient extra
> >    functionality over PEP 273, that it deserves a PEP even now.
> >    But if Just takes time out to write a PEP, I suspect that
> >    makes it even less likely we'll get working code in time...)
>=20
> Indeed. It appears that, if Just completes the patch, we need BDFL
> pronouncement, anyway, to chose one of the following options:

Agreed.

BTW, has all the discussion around Just's proposal diverted anyone
from trying Jim's patch? I'd like to see some more success reports
from using it. I'd hate to have Guido pronounce in favour of Jim's
patch only to discover problems in it...

Paul.


From just@letterror.com  Fri Dec  6 11:34:06 2002
From: just@letterror.com (Just van Rossum)
Date: Fri,  6 Dec 2002 12:34:06 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-ABEF9E22090E11D7AD06003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> From: Martin v. L=F6wis [mailto:martin@v.loewis.de]
> > __path__[0] cannot be an object originally, because the
> > package loader inserts a string, always.
>=20
> Thinking some more, I'd dispute that "always". The *current*
> package loader inserts a string, yes. Under Just's patch,
> I think this remains true, as well.

The last patch I posted here inserts zipimporter instances in __path__, j=
ust as
it replaces *.zip sys.path entries with zipimporter instances. Allowing o=
bjects
on sys.path/pkg.__path__ is still the most lo-tech solution, but it seems
there's too much opposition for that, so I'll continue to work on an
iu.py-inspired solution. I hope to post a patch later today. (Since I don=
't have
access to a Windows box I can't test on windows, so any help from your en=
d (or
anyone's) will be greatly appreciated.)

Just


From martin@v.loewis.de  Fri Dec  6 11:44:11 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 12:44:11 +0100
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com>
Message-ID: <m31y4vjrsk.fsf@mira.informatik.hu-berlin.de>

"Moore, Paul" <Paul.Moore@atosorigin.com> writes:

> That would result in things like ".../lib/package.zip/plat-win32"
> being on __path__. Will Just's code handle this?

Good question. I could not make this work, but I agree it is essential
to package the standard library in a zip file. Just?

Regards,
Martin


From pedronis@bluewin.ch  Fri Dec  6 11:46:28 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Fri, 6 Dec 2002 12:46:28 +0100
Subject: [Python-Dev] Another approach for the import mechanism
References: <16E1010E4581B049ABC51D4975CEDB88619978@UKDCX001.uk.int.atosorigin.com> <m31y4vjrsk.fsf@mira.informatik.hu-berlin.de>
Message-ID: <005901c29d1d$1d795ae0$6d94fea9@newmexico>

From: "Martin v. L=F6wis" <martin@v.loewis.de>
> "Moore, Paul" <Paul.Moore@atosorigin.com> writes:
>
> > That would result in things like ".../lib/package.zip/plat-win32"
> > being on __path__. Will Just's code handle this?
>
> Good question. I could not make this work, but I agree it is essential
> to package the standard library in a zip file. Just?

Btw Jython handle them.

regards.



From guido@python.org  Fri Dec  6 12:36:05 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 07:36:05 -0500
Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS
In-Reply-To: Your message of "Thu, 05 Dec 2002 19:41:18 -0200."
 <20021205194118.C18494@ibook.distro.conectiva>
References: <15852.57674.16727.930806@montanaro.dyndns.org> <asip5f$9uf$1@main.gmane.org>
 <20021205194118.C18494@ibook.distro.conectiva>
Message-ID: <200212061236.gB6Ca5Y15923@pcp02138704pcs.reston01.va.comcast.net>

> > Feel free to only edit Misc/NEWS; before releases I'll go through it
> > to find things I've missed, so there's no need to update whatsnew23.tex.
> > I don't mind if people edit whatsnew23.tex to add items, but it's 
> > certainly not necessary.
> 
> Thanks for mentioning that. I was really wondering about what to do.

Someone should add this to the developer guidelines on python.org/dev/

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Fri Dec  6 12:52:58 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 13:52:58 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <02ac01c29d26$681e00b0$0900a8c0@spiff>

Just van Rossum wrote:

> It seems the majority doesn't like non-strings on sys.path, which to =
me is
> understandable, especially if you take PYTHONPATH into consideration. =
Let's try
> to focus on that.

Has anyone presented any hard data, or are people just guessing?  As far
as I can tell, I'm the only one here who has posted fragile code from a =
real
shipping product -- and I still believe it's a bogus argument.

A quick look in our code repository indicates that most common sys.path
operation is insert/append, followed by not using it at all, followed by =
not
using it at all, followed by not using it at all, followed by =
calculating a list
of dictionaries and *replacing* the current sys.path, followed by =
looping
over sys.path to scan for data or plugin modules (most such code is al-
ready safe, none will break "silently" if a non-string object would =
appear
on sys.path out of the blue).  Compared to the append changes (lots of
work), moving from exception strings to instances (caused some of the
trickiest bugs I've ever seen), and other non-compatible changes, the
impact here is close to zero.

I can dig out more detailed data if necessary (but not today).

Anyone else?

</F>



From fredrik@pythonware.com  Fri Dec  6 13:32:59 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 14:32:59 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]> <02ac01c29d26$681e00b0$0900a8c0@spiff>
Message-ID: <036701c29d2b$ff9d5210$0900a8c0@spiff>

> using it at all, followed by not using it at all, followed by =
calculating a list
> of dictionaries and *replacing* the current sys.path

dictionaries =3D> directories

</F>



From jacobs@penguin.theopalgroup.com  Fri Dec  6 13:32:21 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 08:32:21 -0500 (EST)
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <02ac01c29d26$681e00b0$0900a8c0@spiff>
Message-ID: <Pine.LNX.4.44.0212060759480.8518-100000@penguin.theopalgroup.com>

On Fri, 6 Dec 2002, Fredrik Lundh wrote:
> Just van Rossum wrote:
> > It seems the majority doesn't like non-strings on sys.path, which to me is
> > understandable, especially if you take PYTHONPATH into consideration. Let's try
> > to focus on that.
> 
> Has anyone presented any hard data, or are people just guessing?  As far
> as I can tell, I'm the only one here who has posted fragile code from a real
> shipping product -- and I still believe it's a bogus argument.

I've reviews several hundred thousand lines of code in our repository, and
here is the frequency breakdown of 588 usages of sys.path:

  84%: sys.path.insert(0, package_path)
  10%: sys.path.append(package_path)
   2%: sys.path.extend(path_list)
   1%: str(package_path) in sys.path
  <1%: random debugging/unit test code that prints sys.path or compares
       sys.path with a previous copy of sys.path.

The remaining ~1% use this fairly hairy idiom to implement a plug-in system:

  # in a package __init__.py
  # ...
  for filename in os.listdir(__path__[0]):
    if not filename.endswith('.py') or filename.startswith('__'):
      continue

    name = filename[:-3]

    plugin = __import__('%s.%s' % (base,name), None, None, [name])

    # ...do something with plugin...

Clearly, the above code can be redone in a much nicer way that doesn't
depend on __path__, the qualified base package name, and __import__.

Other than the above hard numbers, I have only some vague user-feelings to
add to the discussion based on a poll of our in-house developers:

  1) They all are very convinced that

       sys.path.append('/usr/path/stuff.zip') 

     should work just like
     
       > mkdir /usr/path/stuff
       > cd /usr/path/stuff
       > unzip -d stuff.zip

       sys.path.append('/usr/path/stuff')

     Any other proposal that magically ignores the zip file name is right
     out, even if it makes it marginally more challenging to wrap the python
     standard library in a zip file.

  2) They don't care if Python stores objects, strings, or bananas in
     sys.path, so long as 

       sys.path=map(str,sys.path) 

     results in a human-readable path and does not change how imports occur.

  3) They all want ZIP archives to be a builtin feature, but don't care if
     they have to write an extra line of code to register a TAR/BZ2/whatever
     import hook.

  4) They don't care if a small amount of code breaks.  It is trivial to
     notice when it does break and easy to fix.


Note that these comments are a result of a 15 minute discussion among a very
mix bag of programmers.  Some have been with Python for many years, others
are in their first week of learning.  Some come from primarily one of
Java/C/C++/VB/Smalltalk/Scheme/Pascal backgrounds, others are extreme language
polyglots.  So, given the amount of discussion on python-dev, I was surprised
how quickly the group converged upon the previous criteria with virtual
unanimity.  The only dissenting voice was our lone .NET fanatic who wouldn't
shut up about how .NET assemblies were by far the superior way of doing
things.  We all have long since learned to ignore him.

Hope this helps,
-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From sholden@holdenweb.com  Fri Dec  6 13:27:59 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Fri, 6 Dec 2002 08:27:59 -0500
Subject: [Python-Dev] Summary for 2002-11-16 through 2002-11-30
References: <Pine.SOL.4.50.0212012051440.29426-100000@death.OCF.Berkeley.EDU>        <20021205173053.A18494@ibook.distro.conectiva>        <Pine.SOL.4.50.0212051620180.28199-100000@death.OCF.Berkeley.EDU> <15855.64475.165789.875723@montanaro.dyndns.org>
Message-ID: <005701c29d2b$4c7a1920$7101a8c0@holdenweb.com>

>
>     Brett> So this brings up something I have been pondering; what to do
>     Brett> when a mistake in a summary is found after it is already out
the
>     Brett> door.
>
> Fix what you can. (tm)
>

...and can what you can't
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Previous .sig file retired to                    www.homeforoldsigs.com
-----------------------------------------------------------------------






From sholden@holdenweb.com  Fri Dec  6 13:31:07 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Fri, 6 Dec 2002 08:31:07 -0500
Subject: [Python-Dev] New and Improved Import Hooks
References: <074b01c29bca$e5298dc0$0201010a@wiktor><016f01c29c99$18967c80$aa2ce8d9@mira><016a01c29ca5$3f8605c0$7101a8c0@holdenweb.com> <m3smxc6qh1.fsf@mira.informatik.hu-berlin.de>
Message-ID: <006901c29d2b$bc51b0a0$7101a8c0@holdenweb.com>

> "Steve Holden" <sholden@holdenweb.com> writes:
>
> > > >Should be "importing_module_from_dlls".
> > >
> > > I still don't understand. Is that some feature of Python?
> > >
> > I think Wiktor is referring to extensions, which under Windows are often
> > provided as .pyd DLLs.
>
> Yes, but then I understand how that is related to zipfile imports: A
> .pyd provides only a single module. A .zip file must supply many
> modules.
>
> > A quick look at my own code base, admittedly puny, reveals no statements
> > that would break were non-strings to be added to sys.path. Is there some
way
> > to trap assignments to sys.path so a warning could be issued if
sufficiently
> > un-stringy elements were discovered in it?
>
> This isn't really possible, as this is just a dict
> modification. You'ld also have to catch assignments to
> sys.__dict__['path']
>

Indeed. But if sys were an instance of some class, rather than a module, we
could trap the assignments in its __setattr__ ... hmm, that's far too
radical. Pity.

regards
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Previous .sig file retired to                    www.homeforoldsigs.com
-----------------------------------------------------------------------





From martin@v.loewis.de  Fri Dec  6 13:51:49 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 06 Dec 2002 14:51:49 +0100
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <Pine.LNX.4.44.0212060759480.8518-100000@penguin.theopalgroup.com>
References: <Pine.LNX.4.44.0212060759480.8518-100000@penguin.theopalgroup.com>
Message-ID: <m3bs3zi7be.fsf@mira.informatik.hu-berlin.de>

Kevin Jacobs <jacobs@penguin.theopalgroup.com> writes:

>   for filename in os.listdir(__path__[0]):
>     if not filename.endswith('.py') or filename.startswith('__'):
>       continue
[...]
> Clearly, the above code can be redone in a much nicer way that doesn't
> depend on __path__, the qualified base package name, and __import__.

That would break only if the package becomes part of a zipfile, right?
And, if it is in a zipfile, it breaks no matter what mechanism we find
for zipfile imports, since it relies on os.listdir to obtain the
package contents, right?

>   1) They all are very convinced that
> 
>        sys.path.append('/usr/path/stuff.zip') 
> 
>      should work just like
>      
>        > mkdir /usr/path/stuff
>        > cd /usr/path/stuff
>        > unzip -d stuff.zip
> 
>        sys.path.append('/usr/path/stuff')

I think both Jim's and Just's patch meet that requirement; Gustavo's
plan doesn't.

>   2) They don't care if Python stores objects, strings, or bananas in
>      sys.path, so long as 
> 
>        sys.path=map(str,sys.path) 
> 
>      results in a human-readable path and does not change how imports occur.

For Just's patch, that is only true for zipfile imports. If you would
also provide URL, or .tgz import with his patch, then you'll loose the
magic when converting the path back to strings.

>   3) They all want ZIP archives to be a builtin feature, but don't care if
>      they have to write an extra line of code to register a TAR/BZ2/whatever
>      import hook.

Clearly, with Jim's patch, they'ld only get zipfile import, period.

> Hope this helps,

It sure does.

Thanks,
Martin


From niemeyer@conectiva.com  Fri Dec  6 13:51:38 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Fri, 6 Dec 2002 11:51:38 -0200
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <m3fztbfyu6.fsf@mira.informatik.hu-berlin.de>
References: <20021205192616.B18494@ibook.distro.conectiva> <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de> <20021205203312.A20315@ibook.distro.conectiva> <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de> <20021205212604.H19843@ibook.distro.conectiva> <m3r8cw5964.fsf@mira.informatik.hu-berlin.de> <20021206005352.C21140@ibook.distro.conectiva> <m3fztbfyu6.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021206115138.B22531@ibook.distro.conectiva>

> Yes, but I don't have to clutter my disk for that.

/usr/lib/python2.2 already exists, doesn't it? That's probably one
of the few __init__.zip we'd ever see, since complete packages
would certainly be a majority. OTOH, with the current implementation,
the standard library is probably the only compressed package we'll
ever see.

That's just my opinion, of course, and I hope to be wrong.

> > - Don't have to change path to use compressed packages (at least
> >   not if you want to provide compressed packages, individual
> >   compressed modules or the standard library).
> 
> I thought you just explained that I will need to change the path to
> provide a compressed standard library, to point to a directory that
> contains an __init__.zip.

No, I didn't. The compressed library would be in /usr/lib/python2.2,
which is already in the path.

> > - Don't have to specify the compression type hardcoded.
> 
> I don't understand that remark. If I have a zipfile, I surely must
> install a zipfile hook in your approach also - a .tar.bz2 hook won't
> be able to load the zipfile, no?

What I mean is:

sys.path = ["/usr/lib/python2.2/stdlib.zip"]

vs.

sys.path = ["/usr/lib/python2.2"]

> > - Allows one to ship a package inside a zip file, without asking
> >   the user to change his path, and without hacking the package.
> > 
> > - Allows one to compress a single file (foobar.py.bz2).
> 
> This is really the same issue: If you had a mechanism to import a
> module from a .py.bz2 file, you could use the same mechanism to import
> a package (or subpackage) from a .zip file. While I think this might
> be desirable, I also think it was never the goal of PEP 273 to provide
> such a facility.

My purpose differs from what is in PEP 273, for sure.

Anyway, I'd just like to expose the idea. If everybody disagrees, we
can safely forget it now.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From fredrik@pythonware.com  Fri Dec  6 13:53:31 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 14:53:31 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <Pine.LNX.4.44.0212060759480.8518-100000@penguin.theopalgroup.com>
Message-ID: <039e01c29d2e$ddddeb00$0900a8c0@spiff>

Kevin wrote:

>   2) They don't care if Python stores objects, strings, or bananas in
>      sys.path, so long as=20
>=20
>        sys.path=3Dmap(str,sys.path)=20
>=20
>      results in a human-readable path and does not change how imports =
occur.

quick question: would the following variation of feeling #2 be =
acceptable:

>   2) They don't care if Python stores objects, strings, or bananas in
>      sys.path, so long as=20
>=20
>        sys.path=3Dmap(str,sys.path)=20

+
+      on a sys.path set up by the Python interpreter itself

>      results in a human-readable path and does not change how imports =
occur.

(note that today's Python already allows you to violate the above rule, =
by
adding non-string objects with cunningly constructed __str__ methods...)

</F>



From jacobs@penguin.theopalgroup.com  Fri Dec  6 13:59:41 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 08:59:41 -0500 (EST)
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <m3bs3zi7be.fsf@mira.informatik.hu-berlin.de>
Message-ID: <Pine.LNX.4.44.0212060853160.8518-100000@penguin.theopalgroup.com>

On 6 Dec 2002, Martin v. [iso-8859-15] L=F6wis wrote:
> Kevin Jacobs <jacobs@penguin.theopalgroup.com> writes:
>=20
> >   for filename in os.listdir(__path__[0]):
> >     if not filename.endswith('.py') or filename.startswith('__'):
> >       continue
> [...]
> > Clearly, the above code can be redone in a much nicer way that doesn'=
t
> > depend on __path__, the qualified base package name, and __import__.
>=20
> That would break only if the package becomes part of a zipfile, right?
> And, if it is in a zipfile, it breaks no matter what mechanism we find
> for zipfile imports, since it relies on os.listdir to obtain the
> package contents, right?

Yup -- which is why I've just added an item to our task queue to avoid th=
e
filesystem entirely for this kind of code.

> >   2) They don't care if Python stores objects, strings, or bananas in
> >      sys.path, so long as=20
> >=20
> >        sys.path=3Dmap(str,sys.path)=20
> >=20
> >      results in a human-readable path and does not change how imports=
 occur.
>=20
> For Just's patch, that is only true for zipfile imports. If you would
> also provide URL, or .tgz import with his patch, then you'll loose the
> magic when converting the path back to strings.

Hmmm.  I haven't looked that closely at Just's proposal.  Why should .tgz=
 or
http://xyz be treated any differently than .zip?

> >   3) They all want ZIP archives to be a builtin feature, but don't ca=
re if
> >      they have to write an extra line of code to register a TAR/BZ2/w=
hatever
> >      import hook.
>=20
> Clearly, with Jim's patch, they'ld only get zipfile import, period.

The sentiment seemed to be that zipfiles were an 80% solution and the res=
t
was icing on the cake.  Of course, as their boss, I can overrule them and
say that I also need an import mechanism that can also gracefully support
cryptographically signed modules (native and in zipfiles) too.  ;)

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From sholden@holdenweb.com  Fri Dec  6 13:56:52 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Fri, 6 Dec 2002 08:56:52 -0500
Subject: [Python-Dev] Re: Doc/whatsnew/* v. Misc/NEWS
References: <15852.57674.16727.930806@montanaro.dyndns.org> <asip5f$9uf$1@main.gmane.org>              <20021205194118.C18494@ibook.distro.conectiva>  <200212061236.gB6Ca5Y15923@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <007d01c29d2f$55d52150$7101a8c0@holdenweb.com>

[amk]
> > > Feel free to only edit Misc/NEWS; before releases I'll go through it
> > > to find things I've missed, so there's no need to update
whatsnew23.tex.
> > > I don't mind if people edit whatsnew23.tex to add items, but it's
> > > certainly not necessary.
> >
[gustavo]
> > Thanks for mentioning that. I was really wondering about what to do.
[guido]
>
> Someone should add this to the developer guidelines on python.org/dev/
>

I added a couple of paragraphs under "Recording Change" in

    http://www.python.org/dev/process.html

I hope they'll serve. Better than nothing, anyhway.

regards
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Previous .sig file retired to                    www.homeforoldsigs.com
-----------------------------------------------------------------------





From jacobs@penguin.theopalgroup.com  Fri Dec  6 14:09:22 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 09:09:22 -0500 (EST)
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <039e01c29d2e$ddddeb00$0900a8c0@spiff>
Message-ID: <Pine.LNX.4.44.0212060900180.8518-100000@penguin.theopalgroup.com>

On Fri, 6 Dec 2002, Fredrik Lundh wrote:
> quick question: would the following variation of feeling #2 be acceptable:
> 
> >   2) They don't care if Python stores objects, strings, or bananas in
> >      sys.path, so long as 
> > 
> >        sys.path=map(str,sys.path) 
> 
> +
> +      on a sys.path set up by the Python interpreter itself
> 
> >      results in a human-readable path and does not change how imports occur.

I suppose, though the motivation behind it is not transparant to me.  There
are many places in Python where a user can do fruity things, and the general
response has been "DON'T DO THAT!".  Is your qualification an attempt to
close such a loophole or is there a realistic use-case that is affected?

For discussion, here is another use-case that I care about (a little):

  Consider a PYTHONPATH includes a mix of filesystem paths, zip files, maybe
  some tgz or tar.bz2 files.

  We would expect that:

    ':'.join(map(str,sys.path)) == os.environ['PYTHONPATH']

  modulo path normalization and any "extra" items that Python decided to
  toss into sys.path.

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From bckfnn@worldonline.dk  Fri Dec  6 14:22:07 2002
From: bckfnn@worldonline.dk (Finn Bock)
Date: Fri, 06 Dec 2002 15:22:07 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <3DF0B28F.2060400@worldonline.dk>

[Just van Rossum]

> This gets nessesarily more complex for packages, as now we also have to cater
> for __path__ items of this form: "/path/to/my/archive.zip/packagedir". In this
> case we need to create a "subimporter" of "/path/to/my/archive.zip" if only
> because we don't want to reread the zip archive's file index. 

and

 > The last patch I posted here inserts zipimporter instances in
 > __path__, just as it replaces *.zip sys.path entries with zipimporter
 > instances. Allowing objects on sys.path/pkg.__path__ is still the most
 > lo-tech solution, but it seems there's too much opposition for that,
 > so I'll continue to work on an iu.py-inspired solution.

You are getting mighty close to what Jython has hardcoded for its .zip 
support <wink>. If the framework you come up with should be usefull from 
jython, then it is very important that it also somehow support cleanup. 
If objects are added to sys.path and __path__ then cleanup is easily 
handled in the __del__ method of the object. If the object is placed on 
some kind of shadowpath then some other cleanup means should be available.

 From jython I can also note that:

- The automatic conversion of a '*.zip' string to a SyspathArchive (aka
   zipimporter) object works, but it works by magic and I'm not happy
   with this kind of identity change. It was just the best I could come
   up with that allowed easy cleanup as well as only-strings-on-sys.path.

- Not all jython's potential importers can be represented by a string.
   Obviously such an importer can't be added on our PYTHONPATH, but must
   be added by program itself. An importer that is based around a java
   classloader is an obvious example of such an importer.

So I would prefer that:

- non-string objects was acceptable values on sys.path. If
   string-subclasses becomes a requirement, that is also fine.

- if strings on sys.path are changed to something else, it should be
   done explicit and not implicit at the time of the first import that
   uses the sys.path entry.


It would also be nice if user code could use the import framework to 
access resources, data files, gifs etc. That would match java's 
ClassLoader.getResourceAsStream() method.


regards,
finn



From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Fri Dec  6 14:26:37 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Fri, 6 Dec 2002 15:26:37 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <Pine.LNX.4.44.0212060853160.8518-100000@penguin.theopalgroup.com>
Message-ID: <000b01c29d33$7d692190$3216e8d9@mira>

> Hmmm.  I haven't looked that closely at Just's proposal.  Why should
.tgz or
> http://xyz be treated any differently than .zip?

There is a magic, in import.c, which, when an import statement is
executed, looks for sys.path entries ending in .zip, and silently
replaces them with zipimport objects. No such magic is available for any
other form of sys.path entries. So while you can add a urlimport object
to sys.path, you lose the hook when you convert the object to a string,
as nobody will recreate the hook.

Regards,
Martin



From guido@python.org  Fri Dec  6 14:21:00 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 09:21:00 -0500
Subject: [Python-Dev] Import: where's the PEP?
In-Reply-To: Your message of "Thu, 05 Dec 2002 23:02:58 EST."
 <20021206040258.GA24214@panix.com>
References: <20021206040258.GA24214@panix.com>
Message-ID: <200212061421.gB6EL0B16459@pcp02138704pcs.reston01.va.comcast.net>

> Having just slogged my way through the entire import thread so far, I've
> got to ask why nobody else has demanded a PEP before we go any further?
> There are too many competing proposals; we need a good summary of what
> design we're planning, plus a summary of why all other proposals are
> rejected.
> 
> I'm a very strong -1 on anything until there's a PEP.

Well, there's PEP 273 which only discusses a feature (import from zip
files) and an API (strings in sys.path that reference zip files).
The PEP stands (I pronounce it accepted if it helps).

But I agree that any API changes beyond that (either in C or in
Python) need more consideration, and a PEP would be fine.

BTW, it's curious to see the discussion follow the path of the sun
around the earth. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From niemeyer@conectiva.com  Fri Dec  6 14:27:01 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Fri, 6 Dec 2002 12:27:01 -0200
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <02ac01c29d26$681e00b0$0900a8c0@spiff>
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]> <02ac01c29d26$681e00b0$0900a8c0@spiff>
Message-ID: <20021206122700.A24104@ibook.distro.conectiva>

> > It seems the majority doesn't like non-strings on sys.path, which to
> > me is understandable, especially if you take PYTHONPATH into
> > consideration. Let's try to focus on that.
> 
> Has anyone presented any hard data, or are people just guessing?  As far
> as I can tell, I'm the only one here who has posted fragile code from a real
> shipping product -- and I still believe it's a bogus argument.

What Just said is that "majority doesn't like", not that there is
breaking code. You can destroy the interpreter without breaking old
code. I, personaly, *don't like* that, since it looks like an ugly hack
to me. You may not care about this, and I don't expect to change the
world with my feelings. Indeed, I've even tried to expose a different
approach, as a way to avoid saying just "I don't like", but it looks
like you don't care anyway, right? My intention was not to just create
another flamewar in python-dev, I can guarant you.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From guido@python.org  Fri Dec  6 14:32:43 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 09:32:43 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Fri, 06 Dec 2002 11:40:50 +0100."
 <025801c29d13$f30f97a0$0900a8c0@spiff>
References: <r01050400-1022-78DEC7E8090311D7AD06003065D5E7E4@[10.0.0.23]>
 <025801c29d13$f30f97a0$0900a8c0@spiff>
Message-ID: <200212061432.gB6EWh616558@pcp02138704pcs.reston01.va.comcast.net>

> > I'm not sure if the two-step approach is a big enough deal to
> > keep it in a new import scheme.
> 
> does anyone remember/know why it was done in two steps?

So you can check whether a module or package exists without importing
it?

For the relative import ambiguity?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jacobs@penguin.theopalgroup.com  Fri Dec  6 14:32:17 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 09:32:17 -0500 (EST)
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <000b01c29d33$7d692190$3216e8d9@mira>
Message-ID: <Pine.LNX.4.44.0212060929240.8518-100000@penguin.theopalgroup.com>

On Fri, 6 Dec 2002, Martin v. L=F6wis wrote:
> > Hmmm.  I haven't looked that closely at Just's proposal.  Why should =
.tgz or
> > http://xyz be treated any differently than .zip?
>=20
> There is a magic, in import.c, which, when an import statement is
> executed, looks for sys.path entries ending in .zip, and silently
> replaces them with zipimport objects. No such magic is available for an=
y
> other form of sys.path entries. So while you can add a urlimport object
> to sys.path, you lose the hook when you convert the object to a string,
> as nobody will recreate the hook.

Don't like that -- a lazy and incremental replacement method seems much m=
ore
sensible, and preserves the semantics that non-python-dev-subscribers (an=
d
I) desire.

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From fredrik@pythonware.com  Fri Dec  6 14:42:25 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 15:42:25 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <Pine.LNX.4.44.0212060853160.8518-100000@penguin.theopalgroup.com> <000b01c29d33$7d692190$3216e8d9@mira>
Message-ID: <047201c29d35$b308c880$0900a8c0@spiff>

Martin wrote:

> There is a magic, in import.c, which, when an import statement is
> executed, looks for sys.path entries ending in .zip, and silently
> replaces them with zipimport objects. No such magic is available for =
any
> other form of sys.path entries.

note that there's nothing that prevent anyone from adding hooks
for this in a later stage.

(on the other hand, does anyone seriously believe that many people
will use tar-based formats to store code packages?  and everyone
who think there's such a thing as a transparent network transport
should check out http://www.bricklin.com/chiapaint.htm ;-)

</F>



From just@letterror.com  Fri Dec  6 14:55:25 2002
From: just@letterror.com (Just van Rossum)
Date: Fri,  6 Dec 2002 15:55:25 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <200212061432.gB6EWh616558@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-C40D72CE092A11D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> > > I'm not sure if the two-step approach is a big enough deal to
> > > keep it in a new import scheme.
> > 
> > does anyone remember/know why it was done in two steps?
> 
> So you can check whether a module or package exists without importing
> it?

I have no idea how useful that is in practice.

> For the relative import ambiguity?

This is not true at find-module-time: if module x.y does "import z" two things
are tried:

    - Find 'z' on x.__path__
      We know up front that *if* that suceeds that the full name is "x.z",
      so you might as well attempt to *load* "x.y".

    if the above failed:
    
    - Find 'z' on sys.path, in which case the full name is "z".

It doesn't matter much, though, in that as long as we don't want to rewrite
import.c as a whole we're stuck with the two-step scheme anyway.

Just


From guido@python.org  Fri Dec  6 14:56:15 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 09:56:15 -0500
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: Your message of "Fri, 06 Dec 2002 13:52:58 +0100."
 <02ac01c29d26$681e00b0$0900a8c0@spiff>
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]>
 <02ac01c29d26$681e00b0$0900a8c0@spiff>
Message-ID: <200212061456.gB6EuGI16697@pcp02138704pcs.reston01.va.comcast.net>

> Has anyone presented any hard data, or are people just guessing?

I found two uses in the Zope 3 codebase that would break for
non-strings.

            cwd = os.getcwd()
            for path in sys.path[1:]: # Yeah, we need to exclude the
            cwd itself
                if path != '' and cwd.startswith(path):
                    name = os.path.join(cwd[len(path)+1:], name)
                    name = os.path.normpath(name)
                    break

        d['PYTHONPATH'] = os.pathsep.join(sys.path)

    def getPythonPath(self):
        'See Zope.App.OFS.ApplicationControl.IRuntimeInfo.IRuntimeInfo'
        return tuple(sys.path)
    # The interface promises a tuple of strings

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Fri Dec  6 15:04:01 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 10:04:01 -0500
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: Your message of "Fri, 06 Dec 2002 11:51:38 -0200."
 <20021206115138.B22531@ibook.distro.conectiva>
References: <20021205192616.B18494@ibook.distro.conectiva> <m3wumo6rl1.fsf@mira.informatik.hu-berlin.de> <20021205203312.A20315@ibook.distro.conectiva> <m3fztc6p8x.fsf@mira.informatik.hu-berlin.de> <20021205212604.H19843@ibook.distro.conectiva> <m3r8cw5964.fsf@mira.informatik.hu-berlin.de> <20021206005352.C21140@ibook.distro.conectiva> <m3fztbfyu6.fsf@mira.informatik.hu-berlin.de>
 <20021206115138.B22531@ibook.distro.conectiva>
Message-ID: <200212061504.gB6F41D16796@pcp02138704pcs.reston01.va.comcast.net>

> Anyway, I'd just like to expose the idea. If everybody disagrees, we
> can safely forget it now.

Amen, brother!

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Fri Dec  6 15:04:34 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 16:04:34 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]>              <02ac01c29d26$681e00b0$0900a8c0@spiff>  <200212061456.gB6EuGI16697@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <049601c29d38$ca818350$0900a8c0@spiff>

Guido wrote:

> > Has anyone presented any hard data, or are people just guessing?
>=20
> I found two uses in the Zope 3 codebase that would break for
> non-strings.
>=20
>             cwd =3D os.getcwd()
>             for path in sys.path[1:]: # Yeah, we need to exclude the =
cwd itself
>                 if path !=3D '' and cwd.startswith(path):
>                     name =3D os.path.join(cwd[len(path)+1:], name)
>                     name =3D os.path.normpath(name)
>                     break
>=20
>         d['PYTHONPATH'] =3D os.pathsep.join(sys.path)
>=20
>     def getPythonPath(self):
>         'See =
Zope.App.OFS.ApplicationControl.IRuntimeInfo.IRuntimeInfo'
>         return tuple(sys.path)
>     # The interface promises a tuple of strings

can Zope users modify the path?  (is this "application code", or is this
part of a package that can be used by applications other than the main
Zope programs?)

</F>



From fredrik@pythonware.com  Fri Dec  6 15:09:03 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 16:09:03 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <Pine.LNX.4.44.0212060900180.8518-100000@penguin.theopalgroup.com>
Message-ID: <049901c29d39$6aff05f0$0900a8c0@spiff>

kevin wrote:

> > quick question: would the following variation of feeling #2 be =
acceptable:
> >=20
> > >   2) They don't care if Python stores objects, strings, or bananas =
in
> > >      sys.path, so long as=20
> > >=20
> > >        sys.path=3Dmap(str,sys.path)=20
> >=20
> > +
> > +      on a sys.path set up by the Python interpreter itself
> >=20
> > >      results in a human-readable path and does not change how =
imports occur.
>=20
> I suppose, though the motivation behind it is not transparant to me.

under the proposed scheme, Python will never mess things up for you by
itself.  by default, and as long as you only use the standard =
mechanisms,
all items in the path behaves like strings, and point to things =
(directories
or archives) in the filesystem.

> There are many places in Python where a user can do fruity things, and
> the general response has been "DON'T DO THAT!".  Is your qualification
> an attempt to close such a loophole or is there a realistic use-case =
that
> is affected?

but what if I *want* to do fruity things?  that's the whole point here:
if the framework delegates imports to objects in the sys.path list, it's
dead simple to experiment.

for some reason, all counter-proposals seem to be obsessed with control
("don't do that" in lower or upper case, "don't write code that I may =
find
ugly", "we'll break your programs in the future if you do that", "if you
cannot think of a string representation, you're not qualified to write
custom importers", "all your ZIP archives must use the same name",
etc).

why are we spending this much time and effort on making things hard
to use?

if I know that I can access all modules needed by my program through
a five-line custom importer, why do I have to invent a string naming
scheme, make sure that scheme is unique so no future version of Python
may mistake it for a builtin importer, register the importer using some
function that I can remember the name and signature of, and pray I've
understood the semantics.

isn't it much easier, and much more obvious for anyone who's survived
Python 101, if I can just write

    sys.path =3D [myimporter()]

and leave early that day.

> For discussion, here is another use-case that I care about (a little):
>=20
>   Consider a PYTHONPATH includes a mix of filesystem paths, zip files, =
maybe
>   some tgz or tar.bz2 files.
>=20
>   We would expect that:
>=20
>     ':'.join(map(str,sys.path)) =3D=3D os.environ['PYTHONPATH']
>=20
>   modulo path normalization and any "extra" items that Python decided =
to
>   toss into sys.path.

the proposed scheme supports this for zip files.

you probably don't want to use tgz or tar.bz2 (no random access), but if =
you
insist, you just need to add 2-3 lines to your startup code to activate =
your
custom importer (Python only reads PYTHONPATH on startup anyway).

</F>



From guido@python.org  Fri Dec  6 15:12:08 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 10:12:08 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Fri, 06 Dec 2002 15:55:25 +0100."
 <r01050400-1022-C40D72CE092A11D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-C40D72CE092A11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212061512.gB6FC8g16874@pcp02138704pcs.reston01.va.comcast.net>

> > > > I'm not sure if the two-step approach is a big enough deal to
> > > > keep it in a new import scheme.
> > > 
> > > does anyone remember/know why it was done in two steps?
> > 
> > So you can check whether a module or package exists without importing
> > it?
> 
> I have no idea how useful that is in practice.

Me neither.

> > For the relative import ambiguity?
> 
> This is not true at find-module-time: if module x.y does "import z"
> two things are tried:
> 
>     - Find 'z' on x.__path__
>       We know up front that *if* that suceeds that the full name is "x.z",
>       so you might as well attempt to *load* "x.y".
> 
>     if the above failed:
>     
>     - Find 'z' on sys.path, in which case the full name is "z".
> 
> It doesn't matter much, though, in that as long as we don't want to
> rewrite import.c as a whole we're stuck with the two-step scheme
> anyway.

The problem with trying to load x.y right away is that then you can't
distinguish between "x.y doesn't exist" (and then you go to step 2)
as opposed to "x.y exists, but it tries to import something else that
raised ImportError" (and then you should simply propagate the
exception).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Fri Dec  6 15:22:13 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 10:22:13 -0500
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: Your message of "Fri, 06 Dec 2002 16:04:34 +0100."
 <049601c29d38$ca818350$0900a8c0@spiff>
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]> <02ac01c29d26$681e00b0$0900a8c0@spiff> <200212061456.gB6EuGI16697@pcp02138704pcs.reston01.va.comcast.net>
 <049601c29d38$ca818350$0900a8c0@spiff>
Message-ID: <200212061522.gB6FMDp16946@pcp02138704pcs.reston01.va.comcast.net>

> >     def getPythonPath(self):
> >         'See Zope.App.OFS.ApplicationControl.IRuntimeInfo.IRuntimeInfo'
> >         return tuple(sys.path)
> >     # The interface promises a tuple of strings
> 
> can Zope users modify the path?  (is this "application code", or is this
> part of a package that can be used by applications other than the main
> Zope programs?)

I dunno.  The interface is not that well documented. :-(

I just show that the assumption of strings (or str-like behavior) is
thoroughly established.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Fri Dec  6 15:23:51 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 10:23:51 -0500
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: Your message of "Fri, 06 Dec 2002 16:09:03 +0100."
 <049901c29d39$6aff05f0$0900a8c0@spiff>
References: <Pine.LNX.4.44.0212060900180.8518-100000@penguin.theopalgroup.com>
 <049901c29d39$6aff05f0$0900a8c0@spiff>
Message-ID: <200212061523.gB6FNp216966@pcp02138704pcs.reston01.va.comcast.net>

> but what if I *want* to do fruity things?  that's the whole point here:
> if the framework delegates imports to objects in the sys.path list, it's
> dead simple to experiment.

That's a good point.  The question then is, if you want to use some
3rd party code that assumes strings, and it breaks as a result, what
is broken?  Your code or the 3rd party code?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From aahz@pythoncraft.com  Fri Dec  6 15:31:26 2002
From: aahz@pythoncraft.com (Aahz)
Date: Fri, 6 Dec 2002 10:31:26 -0500
Subject: [Python-Dev] Import: where's the PEP?
In-Reply-To: <200212061421.gB6EL0B16459@pcp02138704pcs.reston01.va.comcast.net>
References: <20021206040258.GA24214@panix.com> <200212061421.gB6EL0B16459@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20021206153125.GA17821@panix.com>

On Fri, Dec 06, 2002, Guido van Rossum wrote:
>Aahz:
>>
>> Having just slogged my way through the entire import thread so far, I've
>> got to ask why nobody else has demanded a PEP before we go any further?
>> There are too many competing proposals; we need a good summary of what
>> design we're planning, plus a summary of why all other proposals are
>> rejected.
>> 
>> I'm a very strong -1 on anything until there's a PEP.
> 
> Well, there's PEP 273 which only discusses a feature (import from zip
> files) and an API (strings in sys.path that reference zip files).
> The PEP stands (I pronounce it accepted if it helps).
> 
> But I agree that any API changes beyond that (either in C or in
> Python) need more consideration, and a PEP would be fine.

Yeah, I should have been clearer that I meant "any changes beyond what's
in PEP 273"; I did read PEP 273 before posting to see whether there was
coverage of the thread.  I think this really needs a new PEP rather than
updating PEP 273, because (as you say) PEP 273 covers its domain clearly.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From Paul.Moore@atosorigin.com  Fri Dec  6 15:33:35 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 6 Dec 2002 15:33:35 -0000
Subject: [Python-Dev] zipimport & import hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861997C@UKDCX001.uk.int.atosorigin.com>

Fredrik Lundh wrote:

> but what if I *want* to do fruity things? that's the whole
> point here: if the framework delegates imports to objects in
> the sys.path list, it's dead simple to experiment.
>
> for some reason, all counter-proposals seem to be obsessed
> with control ("don't do that" in lower or upper case, "don't
> write code that I may =3D find ugly", "we'll break your programs
> in the future if you do that", "if you cannot think of a
> string representation, you're not qualified to write custom
> importers", "all your ZIP archives must use the same name",
> etc).
>
> why are we spending this much time and effort on making things
> hard to use?
>
> if I know that I can access all modules needed by my program
> through a five-line custom importer, why do I have to invent
> a string naming scheme, make sure that scheme is unique so
> no future version of Python may mistake it for a builtin
> importer, register the importer using some function that I can
> remember the name and signature of, and pray I've understood
> the semantics.

Having listened to the arguments, and tried a few things out
myself with iu.py, I now agree with this. Importer objects on
sys.path are *very* easy to use, and nice and clean. I have no
real code myself which will break - and I concede that if others
do, they can speak for themselves.

In fact, I've gone to the other extreme. I think I'd rather see
general objects on sys.path than string subclasses. Basically,
the breakage happens sooner if someone *does* assume that sys.path
items are strings.

I'll drop out of any more discussion on this issue now...
Paul.


From Paul.Moore@atosorigin.com  Fri Dec  6 15:35:45 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 6 Dec 2002 15:35:45 -0000
Subject: [Python-Dev] New and Improved Import Hooks
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861997B@UKDCX001.uk.int.atosorigin.com>

> It doesn't matter much, though, in that as long as we
> don't want to rewrite import.c as a whole we're stuck
> with the two-step scheme anyway.

I don't wish to seem difficult here, but didn't this start
from the basis that import.c was a mess, Jim Ahlstrom's
patch was going to make it worse, and we should clean it up
rather than making it worse?

Don't get me wrong, I'd like to see a better import hook
mechanism. And I'd like to see zipfile imports. And I'd
like to see the import code cleaned up.

But iu.py does all I need from import hooks. Jim Ahlstrom's
patch does zip imports. If Just's code doesn't clean up the
import code, I quite honestly don't understand the point.

I think this reinforces the need for a PEP. It would clearly
describe the benefits of the patch - both in overall terms,
and in comparison to the (more limited) scope of PEP 273.

Paul.


From guido@python.org  Fri Dec  6 15:46:07 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Dec 2002 10:46:07 -0500
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: Your message of "Fri, 06 Dec 2002 15:33:35 GMT."
 <16E1010E4581B049ABC51D4975CEDB8861997C@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB8861997C@UKDCX001.uk.int.atosorigin.com>
Message-ID: <200212061546.gB6Fk7J17128@pcp02138704pcs.reston01.va.comcast.net>

> Having listened to the arguments, and tried a few things out
> myself with iu.py, I now agree with this. Importer objects on
> sys.path are *very* easy to use, and nice and clean. I have no
> real code myself which will break - and I concede that if others
> do, they can speak for themselves.
> 
> In fact, I've gone to the other extreme. I think I'd rather see
> general objects on sys.path than string subclasses. Basically,
> the breakage happens sooner if someone *does* assume that sys.path
> items are strings.

This is a convincing argument.  Maybe we should just start warning
that sys.path may contain objects in the future.

The str subclass hack still smells ugly to me.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Fri Dec  6 15:54:42 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Fri, 06 Dec 2002 16:54:42 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <r01050400-1022-13B2F945090A11D7AD06003065D5E7E4@[10.0.0.23]> <02ac01c29d26$681e00b0$0900a8c0@spiff>
Message-ID: <3DF0C842.1090408@lemburg.com>

Fredrik Lundh wrote:
> Just van Rossum wrote:
> 
> 
>>It seems the majority doesn't like non-strings on sys.path, which to me is
>>understandable, especially if you take PYTHONPATH into consideration. Let's try
>>to focus on that.
> 
> Has anyone presented any hard data, or are people just guessing?  As far
> as I can tell, I'm the only one here who has posted fragile code from a real
> shipping product -- and I still believe it's a bogus argument.

Have a look at distutils/command/install.py for an example:

         sys_path = map(os.path.normpath, sys.path)
         sys_path = map(os.path.normcase, sys_path)
         install_lib = os.path.normcase(os.path.normpath(self.install_lib))
         if (self.warn_dir and
             not (self.path_file and self.install_path_file) and
             install_lib not in sys_path):
             self.warn(("modules installed to '%s', which is not in " +
                        "Python's module search path (sys.path) -- " +
                        "you'll have to change the search path yourself") %
                       self.install_lib)

Most code we use is of the form:

	sys.path.append('dir')
	sys.path.insert(0, 'dir')
	sys.path = ['/usr/lib/python1.5/site-packages'] + sys.path

How I wish that we had some kind of code base archive for
third party code... e.g. the compile farm would probably be
great for this.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From fredrik@pythonware.com  Fri Dec  6 16:18:07 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 6 Dec 2002 17:18:07 +0100
Subject: [Python-Dev] zipimport & import hooks
References: <Pine.LNX.4.44.0212060900180.8518-100000@penguin.theopalgroup.com>              <049901c29d39$6aff05f0$0900a8c0@spiff>  <200212061523.gB6FNp216966@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <051501c29d43$1158b0f0$0900a8c0@spiff>

Guido wrote:

> > but what if I *want* to do fruity things?  that's the whole point =
here:
> > if the framework delegates imports to objects in the sys.path list, =
it's
> > dead simple to experiment.
>=20
> That's a good point.  The question then is, if you want to use some
> 3rd party code that assumes strings, and it breaks as a result, what
> is broken?  Your code or the 3rd party code?

PIL used to assume strings, and there are a few bug reports on this
topic in the PIL issue database.

they're all very polite, so I think the answer might be "I did something
fruity, so it's probably my fault, and I'm prepared to work around the
problem, but it would be nice if the 3rd party library was fixed too".

or in other words, we should warn application developers that explicitly
adding non-string objects to the path may cause existing code (including
3rd party libraries) to break, and encourage 3rd party developers to =
write
robust code.

</F>



From mclay@nist.gov  Fri Dec  6 16:38:29 2002
From: mclay@nist.gov (Michael McLay)
Date: Fri, 6 Dec 2002 11:38:29 -0500
Subject: [Python-Dev] PEP 242 Numeric kinds
In-Reply-To: <Pine.SOL.4.50.0212052304510.28199-100000@death.OCF.Berkeley.EDU>
References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> <Pine.SOL.4.50.0212052304510.28199-100000@death.OCF.Berkeley.EDU>
Message-ID: <200212061138.30065.mclay@nist.gov>

On Friday 06 December 2002 02:12 am, Brett Cannon wrote:
> [Paul F Dubois]
>
> > We had put off consideration of PEP 242 Numeric kinds until June 2002
> > when a meeting of some interested parties was to occur but the meeting
> > didn't occur. 
[...] 
> I just want to be able to add up a bunch of money values and not have a
> few pennies disappear in the end; usually I am the one who has to eat the
> loss and I am an poor college graduate who can't afford the rounding.  =)

Does the kinds implementation work with binary floats and decimal numbers? 
>From a quick scan of the PEP it looks like it is only binary floats. If it 
only applies to binary numbers then the kinds capability will not eliminate 
decimal number rounding errors. That problem will require extending Python to 
include a decimal number type. 

The abstract mentions a clarification of decimal literals, but I suspect Paul 
was not suggesting that the numbers defined by kinds will use decimal 
arithmetic instead of binary arithmetic.


From bbum@codefab.com  Fri Dec  6 16:04:31 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Fri, 6 Dec 2002 11:04:31 -0500
Subject: [Python-Dev] Re: Patched Transport
In-Reply-To: <011501c29d08$eab820f0$0900a8c0@spiff>
Message-ID: <67C7F84A-0934-11D7-B650-000393877AE4@codefab.com>

On Friday, December 6, 2002, at 04:21 AM, Fredrik Lundh wrote:

> hi bill,
>
>
>> As I move to a Transport subclass that uses urllib2, I first created a
>> Transport subclass whose request() method is patched to delegate all
>> interaction with the connection to methods on the Transport class.
>
> looks fine.
>
> just one nit: changing the make_connection signature may break
> existing programs.  how about adding yet another hook:
>
>     h = self.make_connection(host)
>     if verbose:
>         self.set_debuglevel(h, verbose)
>
>     ...
>
>     def set_debuglevel(self, connection, verbosity):
>         ...
>
> cheers /F

I made the change as recommended -- I was being lazy...

XMLRPC proxying now works and, along with it, so does ProxyHandler 
ordering.    The fix I have does not modify urllib2 -- for my current 
needs, I'm trying to stick with stock python as much as possible.  As 
such, I created a _fixUpHandlers() that will fix the handler order of 
any Opener instance -- likely, it should be eliminated and a similar 
fix integrated into add_handler() on Opener directly.

I will test to make sure it works with a mix of HTTPS and with proxy 
servers that require authentication [I'm currently testing without a 
network connection on my laptop while on the train -- thankfully, OS 
X's apache configuration includes mod_proxy in the apache 
configuration... just have to turn it on].

Caveats are noted in the code.   In particular, using urllib2 really 
wants the full URL so that it can do protocol resolution internally.   
I also ended up with a dummy class (_TransportConnection) that can 
cache-and-carry the hunks of information passed to the various send_* 
and get_* methods as the granularity of those methods simply does not 
match the calling semantics of the urllib2 library.

To initialize a ServerProxy with HTTPTransport, I use the following 
function.  Note that the arguments passed in are identifical to the 
initializer for ServerProxy.   proxyUrl, proxyUser, proxyPass are 
currently stored as module variables -- this will change in the very 
near future, but is largely irrelevant in the context of xmlrpclib / 
urllib2.  Of relevancy, I believe that proxyUrl could be of the form 
http://<user>:<pass>@host:port/path/ and it would be no different than 
creating a HTTPTransport by passing in the user/pass seperately.

def serverProxyForUrl(uri, transport=None, encoding=None, verbose=0):
     if not transport:
         transport = HTTPTransport.HTTPTransport(uri, proxyUrl, 
proxyUser, proxyPass)
     return  xmlrpclib.ServerProxy(uri, transport, encoding, verbose)

(4 space indent, not tabs-- I believe this is the recommended standard 
for the python library?)

In operation, HTTPTransport could completely replace the Transport and 
SafeTransport classes found within xmlrpclib.  However, the one 
incompatible change is in the way the transport class is instantiated.  
The __init__ method requires a single argument;   the full URI of the 
XML-RPC server.   ServerProxy's __init__() could easily be modified to 
address this issue.

Next, I'll probably and compressed content support -- in my app, I can 
reduce the bytecount sent between client and server by upwards of 90% 
by simply gzip'ing the XML prior to dropping it on the wire.  I haven't 
even remotely looked at how to do this.  The actually 
compression/decompression is trivial, but there are a lot of possible 
spots to stick the compression/decompression mechanism.

b.bum

---

"""
HTTPTransport provides an urllib2 based communications channel for use 
in xmlrpclib.

Created by Bill Bumgarner <bbum@mac.com>.

Using HTTPTransport allows the XML-RPC library to take full advantage 
of the features perpetuated by urllib2, including HTTP proxy support 
and a number of different authentication schemes.  If urllib2 does not 
provide a handler capable of meeting the developer's need, the 
developer can create a custom Handler without requiring any changes to 
the XML-RPC code.

Example usage:

def serverProxyForUrl(uri, transport=None, encoding=None, verbose=0):
     if not transport:
         transport = HTTPTransport.HTTPTransport(uri, proxyUrl, 
proxyUser, proxyPass)
     return  xmlrpclib.ServerProxy(uri, transport, encoding, verbose)
"""
import xmlrpclib
from xmlrpclib import ProtocolError
from urllib import splittype, splithost
import urllib2
import sys

class _TransportConnection:
     pass

def _fixHandlerArrayOrder(handlers):
     insertionPoint = 0
     for handlerIndex in range(0, len(handlers)):
         aHandler = handlers[handlerIndex]
         if isinstance(aHandler, urllib2.ProxyHandler):
             del handlers[handlerIndex]
             handlers.insert(insertionPoint, aHandler)
             insertionPoint = insertionPoint + 1


def _fixUpHandlers(anOpener):
     ### Moves proxy handlers to the front of the handlers in anOpener
     #
     # This function preserves the order of multiple proxyhandlers, if 
present.
     # This appears to be wasted effort in that build_opener() chokes if 
there
     # is more than one instance of any given handler class in the 
arglist.
     _fixHandlerArrayOrder(anOpener.handlers)
     map(lambda x: _fixHandlerArrayOrder(x), 
anOpener.handle_open.values())

class HTTPTransport(xmlrpclib.Transport):
     """Handles an HTTP transaction to an XML-RPC server using urllib2 
[eventually]."""
     def __init__(self, uri, proxyUrl=None, proxyUser=None, 
proxyPass=None):
         ### this is kind of nasty.  We need the full URI for the 
host/handler we are connecting to
         # to properly use urllib2 to make the request.  This does not 
mesh completely cleanly
         # with xmlrpclib's initialization of ServerProxy.
         self.uri = uri
         self.proxyUrl = proxyUrl
         self.proxyUser = proxyUser
         self.proxyPass = proxyPass

     def request(self, host, handler, request_body, verbose=0):
         # issue XML-RPC request

         h = self.make_connection(host)
         self.set_verbosity(h, verbose)

         self.send_request(h, handler, request_body)
         self.send_host(h, host)
         self.send_user_agent(h)
         self.send_content(h, request_body)

         errcode, errmsg, headers = self.get_reply(h)

         if errcode != 200:
             raise ProtocolError(
                 host + handler,
                 errcode, errmsg,
                 headers
                 )

         self.verbose = verbose

         return self.parse_response(self.get_file(h))

     def make_connection(self, host, verbose=0):
         return  _TransportConnection()

     def set_verbosity(self, connection, verbose):
         connection.verbose = verbose

     def send_request(self, connection, handler, request_body):
         connection.request = urllib2.Request(self.uri, request_body)

     def send_host(self, connection, host):
         connection.request.add_header("Host", host)

     def send_user_agent(self, connection):
         # There is no way to override the 'user-agent' sent by the 
UrlOpener.
         # This will cause a second User-agent header to be sent.
         # This is both different from the urllib2 documentation of 
add_header()
         # and would seem to be a bug.
         #
         # connection.request.add_header("User-agent", self.user_agent)
         pass

     def send_content(self, connection, request_body):
         connection.request.add_header("Content-Type", "text/xml")

     def get_reply(self, connection):
         proxyHandler = None
         if self.proxyUrl:
             if self.proxyUser:
                 type, rest = splittype(self.proxyUrl)
                 host, rest = splithost(rest)

                 if self.proxyPass:
                     user = "%s:%s" % (self.proxyUser, self.proxyPass)
                 else:
                     user = self.proxyUser

                 uri = "%s://%s@%s%s" % (type, user, host, rest)
             else:
                 uri = self.proxyUrl
             proxies = {'http':uri, 'https':uri}
             proxyHandler = urllib2.ProxyHandler(proxies)

         opener = urllib2.build_opener(proxyHandler)
         _fixUpHandlers(opener)
         try:
             connection.response = opener.open(connection.request)
         except urllib2.HTTPError, c:
             return c.code, c.msg, c.headers

         return 200, "OK", connection.response.headers

     def get_file(self, connection):
         return connection.response




From just@letterror.com  Fri Dec  6 15:48:17 2002
From: just@letterror.com (Just van Rossum)
Date: Fri,  6 Dec 2002 16:48:17 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8861997B@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-89F3F6EA093A11D7AD06003065D5E7E4@[10.0.0.23]>

Moore, Paul wrote:

> But iu.py does all I need from import hooks. Jim Ahlstrom's
> patch does zip imports. If Just's code doesn't clean up the
> import code, I quite honestly don't understand the point.

I won't clean up import.c, but I don't want to make it any worse, which is IMO
what the current zip import patch does. I *might* open doors for future
cleansing, but that's all. My main motivation are still:

  - the mechanics of reading zip files should _not_ be in import.c
  - ditto for a directory caching scheme
  - to try to minimally patch import.c so that zipimport can relatively
    cleanly be plugged in (if this leads to a new & approved import
    scheme, that's merely a nice side effect)

Just


From gmcm@hypernet.com  Fri Dec  6 17:11:32 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Fri, 6 Dec 2002 12:11:32 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <025801c29d13$f30f97a0$0900a8c0@spiff>
Message-ID: <3DF093F4.13309.6D603860@localhost>

On 6 Dec 2002 at 11:40, Fredrik Lundh wrote:

[find_module, load_module]

> does anyone remember/know why it was done in two
> steps? 
 
When Greg Stein was arguing for a one-step get_code,
I believe Guido's response to the same question was
"it seemed logical at the time".

-- Gordon
http://www.mcmillan-inc.com/



From gmcm@hypernet.com  Fri Dec  6 17:11:32 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Fri, 6 Dec 2002 12:11:32 -0500
Subject: [Python-Dev] Another approach for the import mechanism
In-Reply-To: <m3adjjjszx.fsf@mira.informatik.hu-berlin.de>
References: <16E1010E4581B049ABC51D4975CEDB88619977@UKDCX001.uk.int.atosorigin.com>
Message-ID: <3DF093F4.4309.6D6038A6@localhost>

On 6 Dec 2002 at 12:18, Martin v. L=F6wis wrote:

> - Wait for YAGNI (yet another great new
> infrastructure), to invoke all hooks on all sys.path
> items; this would give backwards compatibility
> (sys.path could continue to be a list of strings),
> extensibility (you could add new strings to
> sys.path, yet the hooks would honor them),
> feature-richness (you can implement a .py.bz2
> importer), and slowness (for invoking every hook on
> every sys.path item). 

A .py.bz2 importer is an example of what Greg 
Stein called a "suffix importer". That's orthogonal
to "path importer"s. I omitted "suffix importer"s from
iu, precisely because of the slowness. If you 
limit yourself to hooking nodes on sys.path (or
on __path__), a pure Python __import__ can
beat the builtin import (if you use efficient
archives).

-- Gordon
http://www.mcmillan-inc.com/



From Chris.Barker@noaa.gov  Fri Dec  6 17:31:50 2002
From: Chris.Barker@noaa.gov (Chris Barker)
Date: Fri, 06 Dec 2002 09:31:50 -0800
Subject: [Python-Dev] Re: [Numpy-discussion] PEP 242 Numeric kinds
References: <000001c29cc3$281ed420$6501a8c0@NICKLEBY> <20021206023728.GA24182@localhost.localdomain>
Message-ID: <3DF0DF06.835F72A3@noaa.gov>

José Fonseca wrote:
> to say that the manipulation of numeric
> arrays is only of interest to scientific programmers is the same of when
> in the early computing days engineers saing that computers would only be
> good for crunching numbers, and that the concept of personal computers
> was just non-sense...

I absolutely concur.
 
> For a non-scientic usage of Numeric see the examples in
> http://www.pygame.org/pcr/repository.php, but I can imagine the
> usefullness of Numeric in many more non-scientific applications:
> imaging, sound visualization plugins, 3D graphics, and probably much
> more.

It goes MUCH farther than this. All these examples are what I would call
serious number crunching. Perhaps not strictly scientific, but certainly
the realm of numeric programming, and all places where the "kinds"
concept would be useful.

However, there is a great deal of usefulness in Numeric for applications
that are not doing a lot of number crunching. Performance is only one
reason to use Numeric. The other reason is much cleaner and easier
syntax for manipulating arrays of numbers, especially ones of more than
one dimension. Being able to slice across multiple dimensions, array
oriented arithmetic, overloaded comparisons, etc. I use Numeric for
virtually every program I write, whether performance is an issue or not.
Not having to write all those ugly, confusing and error prone loops is a
godsend.

All that being said, the "kinds" concept is probably mostly of concern
to the Number crunching community. However, it is also sort of a low
level concept, and having it part of core language makes sense, just
like it makes sense to have complex number and long integers in the core
language.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov


From paul@pfdubois.com  Fri Dec  6 18:44:46 2002
From: paul@pfdubois.com (paul@pfdubois.com)
Date: Fri, 6 Dec 2002 11:44:46 -0700
Subject: [Python-Dev] PEP 242 Numeric kinds
In-Reply-To: <200212061138.30065.mclay@nist.gov>
Message-ID: <3DED0AB000003A8F@mta8.wss.scd.yahoo.com>

PEP-242 does not have anything to do with:
   a. Whether or not some future issue of Numeric goes into the core
   b. Decimal arithmetic
   c. Rational arithmetic

THe current implementation lives in the optional Packages area of the Num=
eric
distribution purely because it was convenient for me to put it there, and=

the most likely customers use Numeric anyway.

The only tie to Numeric is the ability of a kind to tell you the "type le=
tter"
that modules array and Numeric accept as types. 

Background: In Numeric there is an issue that if a Numeric.array a has ty=
pe
'f', corresponding to C 'float' rather than C 'double', it is inconvenien=
t
to 
maintain that type at 'f' during operations with scalars. It  used to be

impossible, now it is inconvenient, and with Numarray it won't be a probl=
em.

Historically then, a lot of us were interested in putting a floating poin=
t
type into Python that corresponded to C 'float'; otherwise something like=

a[i] * b was leading to a double result because a[i], a Python scalar, is=

double. If we had an 'f' scalar, we could make a[i] return that instead
of a 'd' scalar. Yes, there would be no literal equivalent (unless we got=

into the usual fistfight over the literals) but that didn't matter so muc=
h.

With Numarray there is no need for this so the interest in making an 'f'
scalar has declined. 

Still, kinds has its uses anyway. It appears that maybe I'm being too shy=

about its virtues. Certainly many of the standard library modules are equ=
ally
obscure and you could view kinds as a wrapping of a standard C header fil=
e
and therefore of sufficient interest. 

So I will restate my position as, I'd like to push 242 forward or backwar=
d.
It shouldn't sit there in limbo. Advice from the PEPczar about process wo=
uld
be welcomed by me.




From tim@zope.com  Fri Dec  6 21:47:09 2002
From: tim@zope.com (Tim Peters)
Date: Fri, 6 Dec 2002 16:47:09 -0500
Subject: [Python-Dev] We got leaks!
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEILECAA.tim@zope.com>

If someone is up for a good weekend puzzle, try this on for size.  I'd like
to multitask this, if possible.

I'm attacking the datetime module in the CVS sandbox, aiming to get it into
good shape for 2.3a1.  There's a pure Python implementation and a pure C
implementation (incomplete but already quite hefty).  You build the latter
like so, for release mode:

    setup.py build_ext -i

or for debug mode:

    setup.py build_ext -i --debug

test_both.py is a test suite.  To test the Python implementation,

    test_both.py

and to test the C implementation,

    test_both.py c

The tests should all pass.

Running this in a loop, I didn't notice any memory growth after a minute of
CPU time.  To be sure, I replaced test_main like so:

def test_main():
    import gc
    r = unittest.TextTestRunner(stream=sys.stdout, verbosity=2)
    s = test_suite()
    while True:
        r.run(s)
        gc.collect()
        print gc.garbage  # this is always an empty list
        print '*' * 10, 'total refs:', sys.gettotalrefcount()

and ran a debug build, watching the printout of total outstanding refs.
Alas, the C implementation shows a very slow leak this way:

********** total refs: 29633
********** total refs: 29643
********** total refs: 29653
********** total refs: 29663
********** total refs: 29673

So the C implementation is leaking 10 references per test cycle.  Fine, so I
screwed up the C code.  However, the Python implementation leaks close to 4x
faster, 37 references per test cycle.  Jeremy suggests I look for the
missing decref in datetime.py <wink>:

********** total refs: 33921
********** total refs: 33958
********** total refs: 33995
********** total refs: 34032
********** total refs: 34069

Likely suspects are "newer stuff":  new-style classes and class methods.
Possibly unittest too, but hard to imagine how.

Even if you can't make time to dig into this, it would help a little to know
what the C and Python leak rates are on a different platform.  Heck, it
would help to know whether the C code even compiles and the tests pass on a
different platform (I'm running Win2K + current CVS Python + MSVC 6 here).



From jacobs@penguin.theopalgroup.com  Fri Dec  6 22:50:41 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 17:50:41 -0500 (EST)
Subject: [Python-Dev] We got leaks!
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHEEILECAA.tim@zope.com>
Message-ID: <Pine.LNX.4.44.0212061736120.20884-100000@penguin.theopalgroup.com>

On Fri, 6 Dec 2002, Tim Peters wrote:
> If someone is up for a good weekend puzzle, try this on for size.  I'd like
> to multitask this, if possible.

I've found the same leak under Linux+GCC 3.2 (Redhat 8) with CVS HEAD
Python, compiled with debugging:

  Testing the Python implementation.
  ********** total refs: 35554
  ********** total refs: 35591
  ********** total refs: 35628
  ********** total refs: 35665
  ********** total refs: 35702

  (leaking 37 refs per run)

  Testing the C implementation.
  ********** total refs: 31262
  ********** total refs: 31272
  ********** total refs: 31282
  ********** total refs: 31292

  (leaking 10 refs per run)

HOWEVER, the leak disapears when Python is compiled normally, without
debugging.  I've also added some instrumentation to the interpreter and have
determined that the leak is not occuring in a garbage collected object
(anything in gc.get_objects()) that exists after the test-suite has been
run.

> Even if you can't make time to dig into this, it would help a little to know
> what the C and Python leak rates are on a different platform.

Yes -- at least on Linux+GCC 3.2 (Redhat 8.0).

> Heck, it would help to know whether the C code even compiles and the tests
> pass on a different platform (I'm running Win2K + current CVS Python +
> MSVC 6 here).

Yes on that too.

I'm going to start a binary search on the test suite to see if I can isolate
a minimal sequence that shows the leak.

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From jeremy@zope.com (Jeremy Hylton)  Fri Dec  6 22:54:20 2002
From: jeremy@zope.com (Jeremy Hylton) (Jeremy Hylton)
Date: Fri, 6 Dec 2002 17:54:20 -0500
Subject: [Python-Dev] We got leaks!
In-Reply-To: <Pine.LNX.4.44.0212061736120.20884-100000@penguin.theopalgroup.com>
References: <BIEJKCLHCIOIHAGOKOLHEEILECAA.tim@zope.com>
 <Pine.LNX.4.44.0212061736120.20884-100000@penguin.theopalgroup.com>
Message-ID: <15857.10908.925878.557147@slothrop.zope.com>

>>>>> "KJ" == Kevin Jacobs <jacobs@penguin.theopalgroup.com> writes:

  KJ> HOWEVER, the leak disapears when Python is compiled normally,
  KJ> without debugging.

Interesting clue.

  KJ> without debugging.  I've also added some instrumentation to the
  KJ> interpreter and have determined that the leak is not occuring in
  KJ> a garbage collected object (anything in gc.get_objects()) that
  KJ> exists after the test-suite has been run.

Yep.  We tried that too :-).

Jeremy



From jacobs@penguin.theopalgroup.com  Fri Dec  6 23:09:48 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 18:09:48 -0500 (EST)
Subject: [Python-Dev] We got leaks!
In-Reply-To: <Pine.LNX.4.44.0212061736120.20884-100000@penguin.theopalgroup.com>
Message-ID: <Pine.LNX.4.44.0212061803330.20884-100000@penguin.theopalgroup.com>

First minimal test case:

  class Leaker(object):
    def __hash__(self): return 1

  for i in range(10):
      hash( Leaker() )
      print '*' * 10, 'total refs:', sys.gettotalrefcount()

In the context of the datetime objects, this leak only affects the Python
version.  It also requires that the Leaker class is a new-style object.

I'm in the process of isolating the other leak that hits both the Python and
C verions of datetime in the test_pickling() function, though I suspect that
may be other incarnations of the above problem.

Fun, fun!

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From tim.one@comcast.net  Fri Dec  6 23:20:42 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Dec 2002 18:20:42 -0500
Subject: [Python-Dev] We got leaks!
In-Reply-To: <Pine.LNX.4.44.0212061803330.20884-100000@penguin.theopalgroup.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEEBDBAB.tim.one@comcast.net>

[Kevin Jacobs]
> First minimal test case:
>
>   class Leaker(object):
>     def __hash__(self): return 1
>
>   for i in range(10):
>       hash( Leaker() )
>       print '*' * 10, 'total refs:', sys.gettotalrefcount()
>
> In the context of the datetime objects, this leak only affects the
> Python version.  It also requires that the Leaker class is a new-style
> object.

Outstanding!  Thanks, Kevin.  I note that this variant shows the same
growth:

x = Leaker()
for i in range(10):
      hash(x)
      print '*' * 10, 'total refs:', sys.gettotalrefcount()

> I'm in the process of isolating the other leak that hits both the
> Python and C verions of datetime in the test_pickling() function,
> though I suspect that may be other incarnations of the above problem.

Possibly.  I'll try to figure out what's going on above first.

> Fun, fun!

The thrill fades a bit after the three hundredth time <wink>.



From jacobs@penguin.theopalgroup.com  Fri Dec  6 23:25:57 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 18:25:57 -0500 (EST)
Subject: [Python-Dev] We got leaks!
In-Reply-To: <Pine.LNX.4.44.0212061803330.20884-100000@penguin.theopalgroup.com>
Message-ID: <Pine.LNX.4.44.0212061820200.20884-100000@penguin.theopalgroup.com>

On Fri, 6 Dec 2002, Kevin Jacobs wrote:
> I'm in the process of isolating the other leak that hits both the Python and
> C verions of datetime in the test_pickling() function, though I suspect that
> may be other incarnations of the above problem.

Here it is:

  import sys
  import cPickle

  dumped = cPickle.dumps(object())
  for i in range(10):
    cPickle.loads(dumped)
    print '*' * 10, 'total refs:', sys.gettotalrefcount()

With output:
  ********** total refs: 10929
  ********** total refs: 10930
  ********** total refs: 10931
  ********** total refs: 10932
  ********** total refs: 10933
  [...]

The leak occurs in cPickle.loads, but not in pickle.loads.  It also does not
depend on binary or non-binary mode of dumps().  It seems to require a
new-style object to occur, but not one that implements a custom hash
function as in the previously reported leak.

This is all I have time for this evening... If the case(s) are not cracked
tomorrow, I'll take a stab at dissecting the responsible C code.

Enjoy!

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From barry@python.org  Fri Dec  6 23:26:40 2002
From: barry@python.org (Barry A. Warsaw)
Date: Fri, 6 Dec 2002 18:26:40 -0500
Subject: [Python-Dev] We got leaks!
References: <Pine.LNX.4.44.0212061803330.20884-100000@penguin.theopalgroup.com>
 <LNBBLJKPBEHFEDALKOLCKEEBDBAB.tim.one@comcast.net>
Message-ID: <15857.12848.917998.329690@gargle.gargle.HOWL>

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

    >> Fun, fun!

    TP> The thrill fades a bit after the three hundredth time <wink>.

You meant the 300th reboot <2000 & 98 winks>
-B


From tim.one@comcast.net  Fri Dec  6 23:32:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Dec 2002 18:32:18 -0500
Subject: [Python-Dev] We got leaks!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEEBDBAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEECDBAB.tim.one@comcast.net>

> First minimal test case:
>
>   class Leaker(object):
>     def __hash__(self): return 1
>
>   for i in range(10):
>       hash( Leaker() )
>       print '*' * 10, 'total refs:', sys.gettotalrefcount()

This one is clearly a missing decref in slot_tp_hash(), leaking a reference
to the integer result object.  If SourceForge wakes up, I've got a fix ready
to check in.



From jacobs@penguin.theopalgroup.com  Fri Dec  6 23:35:53 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 18:35:53 -0500 (EST)
Subject: [Python-Dev] We got leaks!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEECDBAB.tim.one@comcast.net>
Message-ID: <Pine.LNX.4.44.0212061833280.20884-100000@penguin.theopalgroup.com>

On Fri, 6 Dec 2002, Tim Peters wrote:

> > First minimal test case:
> >
> >   class Leaker(object):
> >     def __hash__(self): return 1
> >
> >   for i in range(10):
> >       hash( Leaker() )
> >       print '*' * 10, 'total refs:', sys.gettotalrefcount()
> 
> This one is clearly a missing decref in slot_tp_hash(), leaking a reference
> to the integer result object.  If SourceForge wakes up, I've got a fix ready
> to check in.

Just caught that too.  FYI, it is also missing from the last Python 2.2
checkout that I did.

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From just@letterror.com  Fri Dec  6 23:55:13 2002
From: just@letterror.com (Just van Rossum)
Date: Sat,  7 Dec 2002 00:55:13 +0100
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: <200212061512.gB6FC8g16874@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-2EF6075E097611D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> The problem with trying to load x.y right away is that then you can't
> distinguish between "x.y doesn't exist" (and then you go to step 2)
> as opposed to "x.y exists, but it tries to import something else that
> raised ImportError" (and then you should simply propagate the
> exception).

Also not true: a function that finds *and* loads a module can return None for
"module not found", a new module object upon succes, and propagate an error if
one did occur during import. I'm tempted to make my new import hook scheme work
that way: it avoids a few hoops I have to go through with import.c, as right now
I'm quite limited what zipimporter.find_module() can communicate to
zipimporter.load_module(). (It would also make it even closer to iu.py ;-)

Just


From Jack.Jansen@oratrix.com  Sat Dec  7 00:17:43 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Sat, 7 Dec 2002 01:17:43 +0100
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <02ac01c29d26$681e00b0$0900a8c0@spiff>
Message-ID: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com>

On vrijdag, dec 6, 2002, at 13:52 Europe/Amsterdam, Fredrik Lundh wrote:

> Just van Rossum wrote:
>
>> It seems the majority doesn't like non-strings on sys.path, which to 
>> me is
>> understandable, especially if you take PYTHONPATH into consideration. 
>> Let's try
>> to focus on that.
>
> Has anyone presented any hard data, or are people just guessing?  As 
> far
> as I can tell, I'm the only one here who has posted fragile code from 
> a real
> shipping product -- and I still believe it's a bogus argument.

There are 3 distinct types of problems in Mac/Lib.
1. macresource locates .rsrc files on sys.path. I think this is a 
similar use case Fredrik presented: finding what is essentially a data 
file on sys.path. This is going to be a real problem if everything is 
packed into a zipfile.
2. buildtools loops over sys.path looking for a specific module (using 
os.path calls). This is a problem, but these uses can be converted to 
case (3).
3. buildtools (in another place) and bundlebuilder use imp.find_module 
to find modules and then proceed to do things like use the resulting 
pathname (like copying the file found). These will probably fail if 
there are zipfiles on sys.path, but at least there is a chance of 
fixing them. imp.find_module should return a recognizable magic cookie 
in stead of the pathname for zipfiles (and, later, for any other 
non-filesystem importers). I think we have to leave it at "magic 
cookie" and declare this Somebody Else's Problem, every time I try to 
think of functionality to put in the cookie I end up with something 
resembling a complete virtual filesystem interface. As a partial fix 
the file object returned by find_module could be an object that would 
"work".
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From DavidA@ActiveState.com  Sat Dec  7 00:33:32 2002
From: DavidA@ActiveState.com (David Ascher)
Date: Fri, 06 Dec 2002 16:33:32 -0800
Subject: [Python-Dev] zipimport & import hooks
In-Reply-To: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com>
References: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com>
Message-ID: <3DF141DC.7090508@ActiveState.com>

A question: Is 'import hook chaining' a requirement?

Should one be able to import from a bzip'ed file inside a zip file at a URL?

A (probably OT) thought: it feels like this should be viewed in the context of 
an even more ambitious (ha!) effort like Tcl's VFS (virtual file system) -- 
see http://mini.net/tcl/vfs.  That is a more ambitious approach, but one that 
I like a lot at least in theory =).

--da



From jacobs@penguin.theopalgroup.com  Sat Dec  7 01:31:45 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Fri, 6 Dec 2002 20:31:45 -0500 (EST)
Subject: [Python-Dev] We got leaks!
In-Reply-To: <Pine.LNX.4.44.0212061820200.20884-100000@penguin.theopalgroup.com>
Message-ID: <Pine.LNX.4.44.0212062023280.12728-200000@penguin.theopalgroup.com>

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to mime@docserver.cac.washington.edu for more info.

--416156269-869349433-1039224705=:12728
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Fri, 6 Dec 2002, Kevin Jacobs wrote:
> Here it is:
> 
>   import sys
>   import cPickle
> 
>   dumped = cPickle.dumps(object())
>   for i in range(10):
>     cPickle.loads(dumped)
>     print '*' * 10, 'total refs:', sys.gettotalrefcount()
> 
> The leak occurs in cPickle.loads, but not in pickle.loads.  It also does not
> depend on binary or non-binary mode of dumps().  It seems to require a
> new-style object to occur, but not one that implements a custom hash
> function as in the previously reported leak.
> 
> This is all I have time for this evening... If the case(s) are not cracked
> tomorrow, I'll take a stab at dissecting the responsible C code.

I lied.  Attached is a patch for this leak and several others in cPickle.c
(the diff around line 3118 to Instance_New is the culprit for the above test
case).  Someone should check over my changes, since I'm getting a little
cross-eyed by now.  This takes care of all leaks from the Python test suite,
though there are still 2 ref-counts leaking per loop when using the C
version of datetime.

Enjoy,
-Kevin

PS: If Tim doesn't apply this patch fairly instantly, I will, of course,
    submit it to sourceforge.  blah... blah... blah...

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com

--416156269-869349433-1039224705=:12728
Content-Type: TEXT/PLAIN; charset=US-ASCII; name="cpickle-refcnt.diff"
Content-Transfer-Encoding: BASE64
Content-ID: <Pine.LNX.4.44.0212062031450.12728@penguin.theopalgroup.com>
Content-Description: 
Content-Disposition: attachment; filename="cpickle-refcnt.diff"

SW5kZXg6IE1vZHVsZXMvY1BpY2tsZS5jDQo9PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09DQpSQ1MgZmlsZTogL2N2c3Jvb3QvcHl0aG9uL3B5dGhvbi9kaXN0L3Ny
Yy9Nb2R1bGVzL2NQaWNrbGUuYyx2DQpyZXRyaWV2aW5nIHJldmlzaW9uIDIu
OTYNCmRpZmYgLXUgLXIyLjk2IGNQaWNrbGUuYw0KLS0tIE1vZHVsZXMvY1Bp
Y2tsZS5jCTE2IFNlcCAyMDAyIDE3OjI2OjIzIC0wMDAwCTIuOTYNCisrKyBN
b2R1bGVzL2NQaWNrbGUuYwk3IERlYyAyMDAyIDAxOjIxOjI4IC0wMDAwDQpA
QCAtMjI1LDEwICsyMjUsNiBAQA0KIAlyZXR1cm4gcjsNCiB9DQogDQotI2Rl
ZmluZSBQREFUQV9BUFBFTkRfKEQsTyxFUikgeyBcDQotICBpZiAoUGRhdGFf
QXBwZW5kKCgoUGRhdGEqKShEKSksIE8pIDwgMCkgcmV0dXJuIEVSOyBcDQot
fQ0KLQ0KICNkZWZpbmUgUERBVEFfQVBQRU5EKEQsTyxFUikgeyAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIFwNCiAgICAgaWYgKCgoUGRhdGEq
KShEKSktPmxlbmd0aCA9PSAoKFBkYXRhKikoRCkpLT5zaXplICYmICAgICAg
ICBcDQogICAgICAgICBQZGF0YV9ncm93KChQZGF0YSopKEQpKSA8IDApICAg
ICAgICAgICAgICAgICAgICAgICAgICAgXA0KQEAgLTg5Nyw2ICs4OTMsNyBA
QA0KIAkJaWYgKGtleSA9PSBOVUxMKQ0KIAkJCXJldHVybiAwOw0KIAkJaWYg
KFB5RGljdF9HZXRJdGVtKHNlbGYtPmZhc3RfbWVtbywga2V5KSkgew0KKwkJ
CVB5X0RFQ1JFRihrZXkpOw0KIAkJCVB5RXJyX0Zvcm1hdChQeUV4Y19WYWx1
ZUVycm9yLA0KIAkJCQkgICAgICJmYXN0IG1vZGU6IGNhbid0IHBpY2tsZSBj
eWNsaWMgb2JqZWN0cyBpbmNsdWRpbmcgb2JqZWN0IHR5cGUgJXMgYXQgJXAi
LA0KIAkJCQkgICAgIG9iai0+b2JfdHlwZS0+dHBfbmFtZSwgb2JqKTsNCkBA
IC05MDQsOSArOTAxLDExIEBADQogCQkJcmV0dXJuIDA7DQogCQl9DQogCQlp
ZiAoUHlEaWN0X1NldEl0ZW0oc2VsZi0+ZmFzdF9tZW1vLCBrZXksIFB5X05v
bmUpIDwgMCkgew0KKwkJCVB5X0RFQ1JFRihrZXkpOw0KIAkJCXNlbGYtPmZh
c3RfY29udGFpbmVyID0gLTE7DQogCQkJcmV0dXJuIDA7DQogCQl9DQorCQlQ
eV9ERUNSRUYoa2V5KTsNCiAJfQ0KIAlyZXR1cm4gMTsNCiB9DQpAQCAtOTE5
LDggKzkxOCwxMCBAQA0KIAkJaWYgKGtleSA9PSBOVUxMKQ0KIAkJCXJldHVy
biAwOw0KIAkJaWYgKFB5RGljdF9EZWxJdGVtKHNlbGYtPmZhc3RfbWVtbywg
a2V5KSA8IDApIHsNCisJCQlQeV9ERUNSRUYoa2V5KTsNCiAJCQlyZXR1cm4g
MDsNCiAJCX0NCisJCVB5X0RFQ1JFRihrZXkpOw0KIAl9DQogCXJldHVybiAx
Ow0KIH0NCkBAIC0zMTE1LDYgKzMxMTYsNyBAQA0KIAkJCVB5X1hERUNSRUYo
c2FmZSk7DQogCQkJcmV0dXJuIE5VTEw7DQogCQl9DQorCQlQeV9ERUNSRUYo
c2FmZSk7DQogCX0NCiANCiAJaWYgKGFyZ3M9PVB5X05vbmUpIHsNCkBAIC0z
MjE4LDcgKzMyMjAsMTAgQEANCiAJaWYgKCFtb2R1bGVfbmFtZSkgIHJldHVy
biAtMTsNCiANCiAJaWYgKChsZW4gPSAoKnNlbGYtPnJlYWRsaW5lX2Z1bmMp
KHNlbGYsICZzKSkgPj0gMCkgew0KLQkJaWYgKGxlbiA8IDIpIHJldHVybiBi
YWRfcmVhZGxpbmUoKTsNCisJCWlmIChsZW4gPCAyKSB7DQorCQkJUHlfREVD
UkVGKG1vZHVsZV9uYW1lKTsNCisJCQlyZXR1cm4gYmFkX3JlYWRsaW5lKCk7
DQorCQl9DQogCQlpZiAoKGNsYXNzX25hbWUgPSBQeVN0cmluZ19Gcm9tU3Ry
aW5nQW5kU2l6ZShzLCBsZW4gLSAxKSkpIHsNCiAJCQljbGFzcyA9IGZpbmRf
Y2xhc3MobW9kdWxlX25hbWUsIGNsYXNzX25hbWUsIA0KIAkJCQkJICAgc2Vs
Zi0+ZmluZF9jbGFzcyk7DQo=
--416156269-869349433-1039224705=:12728--


From tim@zope.com  Sat Dec  7 03:27:10 2002
From: tim@zope.com (Tim Peters)
Date: Fri, 6 Dec 2002 22:27:10 -0500
Subject: [Python-Dev] We got leaks!
In-Reply-To: <Pine.LNX.4.44.0212062023280.12728-200000@penguin.theopalgroup.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEENDBAB.tim@zope.com>

[Kevin Jacobs]
> ...
> Attached is a patch for this leak and several others in cPickle.c
> ...

Thanks, Kevin!  THe patch was spot-on and I checked it in (and marked as a
bugfix candidate since the same cPickle leaks appear to exist in 2.2.2).

I already backported the tp_slot_hash leak fix to the 2.2 maintenance
branch.

I checked in the leak-detector code to test_both.py, so this will be easier
to try the next time around.

The remaining leaks in the C implementation were my errors, in the date and
datetime pickle support functions.

There are no visible leaks remaining in the Python or C implementations of
the sandbox datetime code.

BTW, running hefty tests in a loop and staring at sys.gettotalrefcount()
each trip is, as I think this exercise demonstrated, a powerful leak
detector.  I think we plugged about 10 slow leaks here, 8 of them
long-standing leaks in Python (of course, whether they're "slow" or "fast"
leaks depends on the app!  for the test suite, they were slow).

Fun, fun <wink>.



From arigo@tunes.org  Sat Dec  7 16:02:18 2002
From: arigo@tunes.org (Armin Rigo)
Date: Sat,  7 Dec 2002 08:02:18 -0800 (PST)
Subject: [Python-Dev] os.path.commonprefix
Message-ID: <20021207160218.1528F49BA@bespin.org>

Hello everybody,

I recently discovered that os.path.commonprefix(list-of-strings) returns
the longest substring that is an initial segment of all the given strings,
and that this has nothing to do with the fact that the strings might be
paths.  I wonder why this has been put in os.path and not in the string
module in the first place.  This location misled me for a long time into
thinking that commonprefix() would return the longest common *path*, a
feature whose interest I could see.  But:

>>> commonprefix(['/home/users/arigo', '/home/users/alal'])
'/home/users/a'

What is the use for such a thing ??

I suggest that a more descriptive comment be added in the documentation.


Armin



From mwh@python.net  Sat Dec  7 16:20:25 2002
From: mwh@python.net (Michael Hudson)
Date: 07 Dec 2002 16:20:25 +0000
Subject: [Python-Dev] os.path.commonprefix
In-Reply-To: Armin Rigo's message of "Sat,  7 Dec 2002 08:02:18 -0800 (PST)"
References: <20021207160218.1528F49BA@bespin.org>
Message-ID: <2m4r9pssvq.fsf@starship.python.net>

Armin Rigo <arigo@tunes.org> writes:

> >>> commonprefix(['/home/users/arigo', '/home/users/alal'])
> '/home/users/a'
> 
> What is the use for such a thing ??

I seem to recall a discussion which decided that there is no use for
such a thing, but that changing it might break code for people who
found one.  ICBW.

> I suggest that a more descriptive comment be added in the documentation.

No argument from here.

Cheers,
M.

-- 
  We did requirements and task analysis, iterative design, and user
  testing. You'd almost think programming languages were an interface
  between people and computers.                    -- Steven Pemberton
          (one of the designers of Python's direct ancestor ABC)


From bh@intevation.de  Sat Dec  7 16:32:32 2002
From: bh@intevation.de (Bernhard Herzog)
Date: 07 Dec 2002 17:32:32 +0100
Subject: [Python-Dev] os.path.commonprefix
In-Reply-To: <20021207160218.1528F49BA@bespin.org>
References: <20021207160218.1528F49BA@bespin.org>
Message-ID: <6qptsdvlgf.fsf@salmakis.intevation.de>

Armin Rigo <arigo@tunes.org> writes:

> >>> commonprefix(['/home/users/arigo', '/home/users/alal'])
> '/home/users/a'
> 
> What is the use for such a thing ??

I suppose you could use it for something like filename completion as
found in many unix shells. That's perhaps the reason why it's in os.path
and not in string.

   Bernhard

-- 
Intevation GmbH                                 http://intevation.de/
Sketch                                 http://sketch.sourceforge.net/
MapIt!                                           http://www.mapit.de/


From skip@pobox.com  Sat Dec  7 16:33:49 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Dec 2002 10:33:49 -0600
Subject: [Python-Dev] os.path.commonprefix
In-Reply-To: <20021207160218.1528F49BA@bespin.org>
References: <20021207160218.1528F49BA@bespin.org>
Message-ID: <15858.8941.920830.141997@montanaro.dyndns.org>

    Armin> I recently discovered that os.path.commonprefix(list-of-strings)
    Armin> returns the longest substring that is an initial segment of all
    Armin> the given strings, and that this has nothing to do with the fact
    Armin> that the strings might be paths.  

:-) I discovered the same thing over a year ago, fixed the problem in CVS,
then got roundly booed because I'd changed the semantics.  I think it broke
some bit of code of Mark Hammond's.  I recently used this function to find
the common prefix in a bunch of email addresses and prefaced it with this
comment:

  # don't be fooled by "os.path." - commonprefix operates char-by-char!
  pfx = os.path.commonprefix(all_addrs)

The Tools/scripts/trace.py file has a component-wise implementation of
commonprefix().

Skip


From skip@pobox.com  Sat Dec  7 16:34:34 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Dec 2002 10:34:34 -0600
Subject: [Python-Dev] os.path.commonprefix
In-Reply-To: <2m4r9pssvq.fsf@starship.python.net>
References: <20021207160218.1528F49BA@bespin.org>
 <2m4r9pssvq.fsf@starship.python.net>
Message-ID: <15858.8986.387636.729697@montanaro.dyndns.org>

    Michael> ICBW.

Nope, you were spot on. 

Skip


From lists@morpheus.demon.co.uk  Sat Dec  7 15:43:52 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Sat, 07 Dec 2002 15:43:52 +0000
Subject: [Python-Dev] VFS (Was: zipimport & import hooks)
References: <4DEC8EB3-0979-11D7-9220-000A27B19B96@oratrix.com> <3DF141DC.7090508@ActiveState.com>
Message-ID: <n2m-g.el8tq1fr.fsf_-_@morpheus.demon.co.uk>

David Ascher <DavidA@ActiveState.com> writes:

> A (probably OT) thought: it feels like this should be viewed in the
> context of an even more ambitious (ha!) effort like Tcl's VFS
> (virtual file system) -- see http://mini.net/tcl/vfs.  That is a
> more ambitious approach, but one that I like a lot at least in
> theory =).

I agree - this is definitely where my thoughts are heading. I
originally started from the idea of Java's JAR file support - packed
archives of code, which were also executable. Java also supports
getting resources out of JAR files.

I later found Tcl's VFS, and Starkits. These are, in effect, the same
concept.

It's worth remembering,though, that all of these things can be done
now, with varying degrees of success, in Python by user-level
code. All core support adds, really, is a solution to the
bootstrapping problem (you can't import zipfile support from a zip
file, so it can't be included in an application packaged as a zip
file).

Another feature I'd find useful is a token (like Perl's __END__) which
tells the parser to stop parsing at that point. It allows data to be
embedded at the end of a main script. As zipfiles can have arbitrary
data added at the beginning, you can then zip up resources and access
them via zipfile.ZipFile(sys.argv[0]), or even
sys.path.append(sys.argv[0]), allowing you to package code
and data together. [Actually, this last approach won't work under PEP
273, as it mandates a file extension of .zip rather than checking the
file signature. Rats.]

But anything like this is way OT for zipfile imports. It almost
certainly needs a PEP if it is to be done right, as simply stealing
another language's feature without thinking it through is not going to
work...

Paul.
-- 
This signature intentionally left blank


From lists@morpheus.demon.co.uk  Sat Dec  7 15:56:00 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Sat, 07 Dec 2002 15:56:00 +0000
Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension
Message-ID: <n2m-g.bs3xq0vj.fsf@morpheus.demon.co.uk>

PEP 273 does not mandate a particular file extension for zip
files. However, both current implementations (Jim Ahlstrom's patch and
Just's import hooks) recognise zip files via the extension ".zip".

I'd like to suggest that this should not be the case, and that
zipfiles should be recognised by the embedded signature, as is done by
zipfile.is_zipfile(). This will (slightly) slow down the initial
checking for zipfiles, as it requires a file read, but if the result
of the check is cached the performance impact will be minimal.

The main advantage of this is in generality. However, I have a
specific usage in mind - namely, concatenating a Python script with
its supporting code as a zip archive. (Zip archives can have arbitrary
data prepended without affecting their validity). Then, simply by
executing

    sys.path.append(sys.argv[0])

the script can access its own embedded modules.

One problem with getting this to work is that at present, the Python
parser will choke on data appended to the script. This can be avoided
under Windows by a system-specific hack, of embedding a ^Z character
in the script (which text-mode I/O treats as EOF) just before the
appended data. In the longer term, a language-mandated "stop parsing
here" token, like Perl's __END__, might be useful. But that's not
something I'm proposing right now (unless an expert in Python's parser
steps up and says "oh, that's easy - I can implement that in a couple
of lines of code" :-))

Any comments?
Paul.
-- 
This signature intentionally left blank


From martin@v.loewis.de  Sat Dec  7 17:26:30 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 07 Dec 2002 18:26:30 +0100
Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension
In-Reply-To: <n2m-g.bs3xq0vj.fsf@morpheus.demon.co.uk>
References: <n2m-g.bs3xq0vj.fsf@morpheus.demon.co.uk>
Message-ID: <m31y4tlozd.fsf@mira.informatik.hu-berlin.de>

Paul Moore <lists@morpheus.demon.co.uk> writes:

> Any comments?

That's quite expensive. Unless I'm mistaken, it causes an additional
stat call per sys.path item (to find out that it is a directory).

If Just's patch is accepted, you get that feature for free

import zipimport,sys
sys.path.append(zipimport.zipimporter(sys.argv[0]))

does what you want. For Jim's patch, I'd say you lose: We should
tighten the PEP saying that only items in sys.path are treated as
zipfiles that have ".zip" in their name (with a yet-to-be-specified
separator for putting directory in the zipfile into sys.path).

Regards,
Martin


From just@letterror.com  Sat Dec  7 17:56:10 2002
From: just@letterror.com (Just van Rossum)
Date: Sat,  7 Dec 2002 18:56:10 +0100
Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension
In-Reply-To: <m31y4tlozd.fsf@mira.informatik.hu-berlin.de>
Message-ID: <r01050400-1022-32AB73ED0A0D11D7AD06003065D5E7E4@[10.0.0.23]>

Martin v. L=F6wis wrote:

> That's quite expensive. Unless I'm mistaken, it causes an additional
> stat call per sys.path item (to find out that it is a directory).
>=20
> If Just's patch is accepted, you get that feature for free
>=20
> import zipimport,sys
> sys.path.append(zipimport.zipimporter(sys.argv[0]))
>=20
> does what you want. For Jim's patch, I'd say you lose: We should
> tighten the PEP saying that only items in sys.path are treated as
> zipfiles that have ".zip" in their name (with a yet-to-be-specified
> separator for putting directory in the zipfile into sys.path).

Note that I'm currently working on a variation of the patch that implemen=
ts a
subset of iu.py (I think I mentioned this a few times already ;-). For ea=
ch path
item, each item in a list of hooks is asked whether it can handle the pat=
h item;
the first one that can is used. This is done *once* per path item, the re=
sults
are cached in a dict (iu's shadowpath): {path_item: path_handler, ...}. T=
his
gives the zipimporter a chance no matter what the extension is and it's s=
till
cheap as it's only done once.

Just


From aahz@pythoncraft.com  Sat Dec  7 20:21:28 2002
From: aahz@pythoncraft.com (Aahz)
Date: Sat, 7 Dec 2002 15:21:28 -0500
Subject: [Python-Dev] Import compromise
Message-ID: <20021207202128.GA28924@panix.com>

Guido, I call for a Pronouncement:

I'm seeing some convergence on basic criteria needed for a Grand Import
Redesign, but I'm not seeing convergence on the actual design itself.
Time is running short for 2.3, and people making hasty decisions make
poor decisions.

OTOH, the complaints raised about the current patch for PEP 273 have
merit, I think.  On the gripping hand, we really want PEP 273 in some
form for 2.3.  So here's my proposed compromise:

Split the PEP 273 patch into two parts.  The first part is the
absolutely critical modifications to import.c, which basically boils
down to making a function call every time you access a '*.zip' string on
sys.path.  Everything else goes into zipimport.c, a compiled-in module
like socket.  (I'd even support making zipimport.c an include file for
import.c)

Yes, this means we lose the caching for non-ZIP files for 2.3 -- I vote
YAGNI.  Yes, this makes more work over the long haul, but it saves stress
and time now.  Yes, it's more work than just accepting the current PEP
273 patch, but I think there are too many good arguments against that.

zipimport will *NOT* be a publicly exposed library for 2.3, possibly
never.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From fredrik@pythonware.com  Sat Dec  7 20:41:19 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Sat, 7 Dec 2002 21:41:19 +0100
Subject: [Python-Dev] Import compromise
References: <20021207202128.GA28924@panix.com>
Message-ID: <026301c29e31$00632980$ced241d5@hagrid>

Aahz wrote:

> Time is running short for 2.3, and people making hasty decisions make
> poor decisions.

speak for yourself.

</F>



From martin@v.loewis.de  Sat Dec  7 20:54:09 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 07 Dec 2002 21:54:09 +0100
Subject: [Python-Dev] Import compromise
In-Reply-To: <20021207202128.GA28924@panix.com>
References: <20021207202128.GA28924@panix.com>
Message-ID: <m3ptsdim8e.fsf@mira.informatik.hu-berlin.de>

Aahz <aahz@pythoncraft.com> writes:

> I'm seeing some convergence on basic criteria needed for a Grand Import
> Redesign, but I'm not seeing convergence on the actual design itself.
> Time is running short for 2.3, and people making hasty decisions make
> poor decisions.

I disagree with this analysis. Time is not running short for Python
2.3: there isn't even a release schedule, yet, and I question promises
that an alpha release will be produced this year.

> Split the PEP 273 patch into two parts.  The first part is the
> absolutely critical modifications to import.c, which basically boils
> down to making a function call every time you access a '*.zip' string on
> sys.path.  Everything else goes into zipimport.c, a compiled-in module
> like socket.  (I'd even support making zipimport.c an include file for
> import.c)

Good plan. Who is going to provide the implementation?

I think Just would be happily drop his implementation if such a change
would magically appear, but I think that is unlikely to happen.

Regards,
Martin


From just@letterror.com  Sat Dec  7 21:05:30 2002
From: just@letterror.com (Just van Rossum)
Date: Sat,  7 Dec 2002 22:05:30 +0100
Subject: [Python-Dev] Import compromise
In-Reply-To: <m3ptsdim8e.fsf@mira.informatik.hu-berlin.de>
Message-ID: <r01050400-1022-A500ED740A2711D7AD06003065D5E7E4@[10.0.0.23]>

Martin v. L=F6wis wrote:

> I think Just would be happily drop his implementation if such a change
> would magically appear, but I think that is unlikely to happen.

Oh, Aahz's proposal is *very* close to what my current implementation loo=
ks
like, except that it exposes a couple of things here and there ;-)

I'll try to post code later today (it works for me now, but I'd like to g=
o
through it one more time, do some more testing), and will then write a sh=
ort
piece about what it does, why I think it's good that it does that, what t=
he
impact is on import.c, etc. etc. (And depending on the size if that piece=
 I
might call it a "draft PEP"...)

Just


From aahz@pythoncraft.com  Sat Dec  7 21:09:05 2002
From: aahz@pythoncraft.com (Aahz)
Date: Sat, 7 Dec 2002 16:09:05 -0500
Subject: [Python-Dev] Import compromise
In-Reply-To: <r01050400-1022-A500ED740A2711D7AD06003065D5E7E4@[10.0.0.23]>
References: <m3ptsdim8e.fsf@mira.informatik.hu-berlin.de> <r01050400-1022-A500ED740A2711D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <20021207210905.GA10103@panix.com>

On Sat, Dec 07, 2002, Just van Rossum wrote:
> Martin v. Löwis wrote:
>> 
>> I think Just would be happily drop his implementation if such a change
>> would magically appear, but I think that is unlikely to happen.
> 
> Oh, Aahz's proposal is *very* close to what my current implementation
> looks like, except that it exposes a couple of things here and there

Would you have any objection to restricting the scope of your work so
that it *only* covers PEP 273?
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From aahz@pythoncraft.com  Sat Dec  7 21:11:38 2002
From: aahz@pythoncraft.com (Aahz)
Date: Sat, 7 Dec 2002 16:11:38 -0500
Subject: [Python-Dev] Import compromise
In-Reply-To: <m3ptsdim8e.fsf@mira.informatik.hu-berlin.de>
References: <20021207202128.GA28924@panix.com> <m3ptsdim8e.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021207211138.GB10103@panix.com>

On Sat, Dec 07, 2002, Martin v. Löwis wrote:
> Aahz <aahz@pythoncraft.com> writes:
>> 
>> I'm seeing some convergence on basic criteria needed for a Grand Import
>> Redesign, but I'm not seeing convergence on the actual design itself.
>> Time is running short for 2.3, and people making hasty decisions make
>> poor decisions.
> 
> I disagree with this analysis. Time is not running short for Python
> 2.3: there isn't even a release schedule, yet, and I question promises
> that an alpha release will be produced this year.

You may be right; however, the tone of the discussion seems to be partly
predicated on a short timeframe.  I'm fairly confident Guido won't hold
2.3 for lack of a new import facility.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From aahz@pythoncraft.com  Sat Dec  7 21:23:26 2002
From: aahz@pythoncraft.com (Aahz)
Date: Sat, 7 Dec 2002 16:23:26 -0500
Subject: [Python-Dev] Proposed changes to linuxaudiodev
In-Reply-To: <20021130164818.GA25894@cthulhu.gerg.ca>
References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca>
Message-ID: <20021207212326.GA13707@panix.com>

On Sat, Nov 30, 2002, Greg Ward wrote:
> On 29 November 2002, Guido van Rossum said:
>>
>> I recommend that you simply check in your new code as ossaudiodev and
>> we'll mark the linuxaudiodev module as deprecated.
> 
> Sounds fine to me.  Should I worry about adding an official
> DeprecationWarning to linuxaudiodev.c?  Or does that wait until 2.4?

Given its lack of official documentation, I'd vote for DeprecationWarning
now.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From lists@morpheus.demon.co.uk  Sat Dec  7 21:34:25 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Sat, 07 Dec 2002 21:34:25 +0000
Subject: [Python-Dev] Import compromise
References: <20021207202128.GA28924@panix.com>
Message-ID: <n2m-g.n0nh344e.fsf@morpheus.demon.co.uk>

Aahz <aahz@pythoncraft.com> writes:

> Split the PEP 273 patch into two parts.  The first part is the
> absolutely critical modifications to import.c, which basically boils
> down to making a function call every time you access a '*.zip'
> string on sys.path.  Everything else goes into zipimport.c, a
> compiled-in module like socket.  (I'd even support making
> zipimport.c an include file for import.c)

Who's going to write this code? I can't - my involvement has been in
updating Jim's patch to make it apply cleanly to current CVS, testing
it a little, and making some bugfix-type changes. I don't feel
qualified to do the major surgery you're suggesting.

Frankly, there are only 2 relevant chunks of code - Jim's patch and
Just's. The disadvantages of Jim's have been spelled out, and remain
true. Its main advantage is that it works now. Just's patch is far
better in theory. It remains to be seen if he can deliver working code
in the necessary timeframe (whatever that is...). Of course, in the
absence of code, we all get to have a lot of fun talking about what
might go wrong if it works the way we assume it does, but we've got to
while away the time somehow :-)

Paul.
-- 
This signature intentionally left blank


From bac@OCF.Berkeley.EDU  Sat Dec  7 22:08:32 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Sat, 7 Dec 2002 14:08:32 -0800 (PST)
Subject: [Python-Dev] Import compromise
In-Reply-To: <n2m-g.n0nh344e.fsf@morpheus.demon.co.uk>
References: <20021207202128.GA28924@panix.com> <n2m-g.n0nh344e.fsf@morpheus.demon.co.uk>
Message-ID: <Pine.SOL.4.50.0212071404310.3581-100000@death.OCF.Berkeley.EDU>

[Paul Moore]

> Of course, in the
> absence of code, we all get to have a lot of fun talking about what
> might go wrong if it works the way we assume it does,

... and drive the person who has to summarize all of this bloody long
thread insane.

At least it helped give me some inspiration (along with one of Martin's
emails) on how to implement ``dummy_threading`` rather easily.

<goes back to ``dummy_threading`` to avoid the hell that will be the next
summary>

-Brett


From fredrik@pythonware.com  Sat Dec  7 23:47:00 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Sun, 8 Dec 2002 00:47:00 +0100
Subject: [Python-Dev] Import compromise
References: <20021207202128.GA28924@panix.com> <m3ptsdim8e.fsf@mira.informatik.hu-berlin.de>
Message-ID: <02ed01c29e4a$f82e7d40$ced241d5@hagrid>

Martin wrote:

> I disagree with this analysis. Time is not running short for Python 2.3

In addition to the schedule issues (and nothing says that the zip importer
has to be in the first alpha), Aahz also ignores the fact that the people
involved has long experience of python code archives (I wrote squeeze
in 1997, and that wasn't my first archiver; Gordon has been shipping his
installer for several years, the mac folks have been using this stuff since
the early days; that's easily 10-15 years of experience of using archive
technology in real applications.  hasty indeed); he also ignores the fact
that we have two working implementations to chose from, each under-
stood by more than one programmer; that we have input from large-scale
Python users; and that this isn't some groundbreaking change to the
Python language, but merely an extension to the way the CPython
implementation uses the sys.path variable.  Etc.

If this isn't enough to get this feature into CPython 2.3, there's something
wrong with the python-dev process.

</F>



From fredrik@pythonware.com  Sat Dec  7 23:49:47 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Sun, 8 Dec 2002 00:49:47 +0100
Subject: [Python-Dev] Import compromise
References: <20021207202128.GA28924@panix.com> <n2m-g.n0nh344e.fsf@morpheus.demon.co.uk> <Pine.SOL.4.50.0212071404310.3581-100000@death.OCF.Berkeley.EDU>
Message-ID: <02f901c29e4b$5561be00$ced241d5@hagrid>

Brett Cannon wrote:

> ... and drive the person who has to summarize all of this bloody long
> thread insane.

maybe you could team up with Ping and turn his zest tool
into a brettbot?

http://www.sims.berkeley.edu/~ping/vertex-2002/zest.html

</F>



From bac@OCF.Berkeley.EDU  Sat Dec  7 23:55:15 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Sat, 7 Dec 2002 15:55:15 -0800 (PST)
Subject: [Python-Dev] Import compromise
In-Reply-To: <02f901c29e4b$5561be00$ced241d5@hagrid>
References: <20021207202128.GA28924@panix.com> <n2m-g.n0nh344e.fsf@morpheus.demon.co.uk>
 <Pine.SOL.4.50.0212071404310.3581-100000@death.OCF.Berkeley.EDU>
 <02f901c29e4b$5561be00$ced241d5@hagrid>
Message-ID: <Pine.SOL.4.50.0212071552480.3581-100000@death.OCF.Berkeley.EDU>

[Fredrik Lundh]

> Brett Cannon wrote:
>
> > ... and drive the person who has to summarize all of this bloody long
> > thread insane.
>
> maybe you could team up with Ping and turn his zest tool
> into a brettbot?
>
> http://www.sims.berkeley.edu/~ping/vertex-2002/zest.html
>

Perhaps, but part of the reason I wanted to do the Summaries was to force
myself to read the emails so as to make sure I learn all the little
nuggests of knowledge embedded in each email.  I just needed a slight
venting.  =)

I will have a look, though.  Thanks, Frederik.

-Brett


From just@letterror.com  Sun Dec  8 00:44:47 2002
From: just@letterror.com (Just van Rossum)
Date: Sun,  8 Dec 2002 01:44:47 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
Message-ID: <r01050400-1022-44A4925E0A4611D7AD06003065D5E7E4@[10.0.0.23]>

--7584767-3623878656-3248300694=:9002
Content-Type: text/plain; Charset=US-ASCII
Content-Transfer-Encoding: 7bit

Here's the current state of my stuff. I'd rather post it now than to fiddle
more: I'm too tired, I'd only break stuff...

I currently build the zipimport module as a builtin to avoids some bootstrapping
pitfalls (*). It should still work as a shared lib *provided* it is available on
sys.path *before* site.py is run. The same goes more or less for zlib unless
you're using uncompressed archives.

The test suite ran fine with the std lib packaged as a zip file (minus the test
suite it, as it globs for *.py in Lib/test/), apart from three tests that do
stuff with module.__file__ (eg. email can't find it's data files ;-).

Highlights:

- zip file names on sys.path work
- zip file names PYTHONPATH work
- sys.path still contains strings
- module.__file__ is still a regular string, looking like this:
    '/path/to/archive.zip/path/to/module.pyc'
- pkg.__path__ still contains a regular string, that looks like this:
    '/path/to/archive.zip/path/to/packagedir/'

I'll write a bit about the mechanics tomorrow, but in the meantime I would like
to see some test results/bug reports/bug fixes from Windows developers. Please
don't start with the full monty (ie. the std lib), try a small test archive
first please ;-)

There's still plenty to do regarding imp module integration, minor things like
reload() not being supported yet, etc. Nevertheless I think the import.c patch
is remarkably small for what it accomplishes. Needless to say, the zipimport
module itself also needs work.

Fun, fun!

Just

*) It's mostly a problem when you're running Python from the source directory,
when site.py adds stuff to sys.path to get a running system: by then it's too
late to find zipimport. It should be less of a problem with a regular install,
but I haven't tried.
--7584767-3623878656-3248300694=:9002
Content-Type: application/octet-stream; Name="zipper3.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; Filename="zipper3.tgz"

H4sIAAAAAAAAA+w8a1fbSLLz1f4VHecEbGPA5pUsXjLHCZCwlyScQGazm+ToCKkN
WoSkkWQSzwz//VZVvyUZyJnHfrjXJyfG3VXd1dXV9eqSfomyjOeb6z/8iZ/hcGv4
dHsbvofDpzvie7ixJb7F5wcAeLq1Dd2b0D4abW49/YFt/5lEqc+sKP2csR/+A3/c
BQdg0+lfQdBf+/lF7j98R9dZmpdrwR8+x3A0HO5sbS3Y/62NrZ3q/m9ub2/+wIZ/
OCUNn//j+/84SoJ4FnLWOZmXl2mydtlpm7aizGdBec2vz3nu9qRFyKeF23bt58Wl
H7uNQXqdRTHHxnYbmFhGAQsu/bzPsrnHv5U8KaI0KT59YXvs185aNg86A4bfqfzu
3I4Bc73PtIjynKXn/+FByYCGKIlKGID5SciKWYb9rL/ebpfzjEM3E0twkP8dZUfy
7zHSVAP4td06mb+jKbzXB5N985P1/Ty4jG74mDEgKfPLy8S/5iydsvKS4yBMAiAR
FtoUeFAAEqGFETRd8ITnfsywh0XJNK1gZDks7ttYYBCQaNllHX+9mJ2vh1EOgGk+
X+8g6u1Y8/dkfgarlwNZq/WweUzMnKb5Vz8PgYNBjNgaU01/wUuvTIMu7pVedW9c
B4xTP/Rg4d51Gs5i3rXmY/2Cx9MBE2NMZ3GMzBq0W6IBEJCBA1h9yaIi84Mr/4Kr
n1JwwoE1FxDk8aTM50BI+zHtPncW+PKSB1fdNOtpJFqzah6wpSo7eu36ktqWMHgJ
/9p1GNoP4sImys8vnN9XX8Oi1wYZAjYfXSRpzkEmLmbXQDjA2UOj7LLSv+IFC/xc
SpFfAoNg11Ggv3IWpslyCT8AMGXXAMviaMoZMDAEBsM2gtSfB7FfFLxYIxGqbQDw
qoXfcMLc3enBSlafl5nnxzFuNa5rCKxtRVPWJYxHe+zth+PjHh4JGmP1uZJv0TPW
7SThtVYhs6b5tt3KeTnLE9Y1HOsJKm/1VoAEtKuMahKsB2yCEDWUM1wXsDHmifgl
lvnoZD7JL7wT0F38bJaBAIuhOu8e7VokrHlEhOd1YHdauD/wWTqZn5Z5lFyQJIFw
ORzq9QBSLnZ1hPOdzL2jty/fHxx2XUCgDEkCLukBJ4X4ow4p6beBT6NfeA1QLE9B
P99jbyYfTyZnr48P3ortPJkf5Ll3yks5E/z+FniCxdCT5rTSVkdtOI1TpimL0+Si
g6TYq4OdbVckQakQRLSlSkMIqWBLS+yRIAYI4Bmq8zd+GVzyQtH0juipMnTRgGJ5
cPoU5df+nJ1rJQ1n5rqttpB11pG89TJdl9BrsOuoX0G9kl5t2ToZQD0lEiAkfW0y
jNyTwJ3Ppp8Mv1dGX3RPH5RQv3oqWhn8LWW0hWe6Ox6LVbRI+YMqQMrhlJcsDYJZ
nvtJQAvqILmSThoFzBn862YD2dUTlurjx4/Mj4uUrf376ESCI/syzTTa7POc+1dj
3flp6wv2nx6c9PQov/3GJsdn0CRHaenFZGNBxMoe28I/4ayLYSSA3G4YQDWt0qJ7
2KUldZVt9czaE6AYreo0nQETgDP0KyqYXzIOLYIGIxRIpRQXhCrKKI5ZwUsBeKvF
/mXM/bxLQmwOlEsWWwFKaHNEKyxrmxRbmSdBNu/CHg+YMGBqiBU2IgzcftmGfs3y
5+GyOxOMAT/kfIRiGPXJcGL0BTUwsZ8Y4gLg0NA3rvcgIfbEuG45M5BmtLPR5VqX
HObptVQI6txWVXkjrFqJXEplfLP17liW7DmaErf+LJ+jzQMtwnx0IUAhhI6bNoXp
WTEv1txjuSYO9z+17QxDHAcPUICDsHQGuj6+Ufay5aKLBc4L7xUvxbnvdlyIjtkw
F/ORUWgn831w8qQ/4oIp6bZ0h4SGGY9Kfl2BF2oY11SzBYIKPZA1f90pUlBqemRx
kaCeJX+0EOzUXAqVjy1PuavauxVHQg8uQYg2tHcfHYNHfYLw20XH8U6lfo+FWYxN
9Ni2Ayc3zsjQ9j5u0ih03I+QCwep5oGQh4Gr3D9oNuu1Ps2BWo/RBeJ3eu5hDEP+
2TTnwsD3ql4SeoQeDoqe9WLfm/xu+TfpLfFnP/OMBw4/tBO+0HXCwykNgbFvMBpI
GKgBirxQQWr9eJ9DIz0aoxIBC9ukBtUqrt2StEpDKTQZ6DlEBm3neANSu7jOqybK
IC0mT26Gsg+o+5u8LbNnt+0FMyDna4tU8orfzjJ1y4qtv3FlJ/M3fpbh/K/94n/4
3CGXpEoYI3m87c2EgUZjwwJpIDSFHe3a2sv9m1gUjwveNOBwLBWnIl+Sao4UCcvt
fdEV+jYqcjQgoHqqzj1JZE2+6yENoI6V6IIXJwS/L6Tc9cq+SKmOyH4PTAR6b2hQ
OJGBtQQnOJCz216rcPekqJY5nIG8q4lcXlvuuTLb7KH/5MczbjvocgR2PQP3MElL
9HV9MH5lycM17HGcdR2IKZl0VYjQGpooOsRLgn1LRj2MdXTB/g4hY32F5MRGJCgs
AphN+FpZcY6ikcNBJRcUfdGW4DvFvhJiCZuwMOBURgAUmtg/4p5wNUUyRg0Gf71N
E1q3HE22PEDEKUHyZ4m4TqpY0l4X85om/70yb63JkXlFziKhf6S1qHBKXGMA/iGl
Ppr1uIwTG/CVtv6uU6N8U8yzyIn8EjDOZyX4DXSW4BwVNEax6ASpo0eanY6z2RA8
z8osKTDjjRhMhTEW2tb0rayI2K24gpAHRiPr+jtP7XqfOKwsqYxsFCb7tAIBsbYH
X6AHjiVh1O0XNaO129OB9gpaMfgf+8Fir9+tBxryejIkHlByUMfHD9YZAGgPZZSH
9KwXag+lbgy2o4hbbkoSj2SEEBtCwQhaq4lQsSdGHmhTTJqzLZ16/NjZTjvHKRSa
dBMrHUguTbyndJHl2J/7IUjIRZSwZIbJe3Dw2Pm8BN8+DYEUXGACzFPOvZkERhTD
t4I0KaNkxrW3rsQfQFRMWXXeb9sPyCqxjtYkuxBrJMQ4qYA7loYVe2+r1ze8vEzD
fT51sqjX1CpvENqtXzuOQWYLvI0Be3Nw9tr7afJ+8v7VKRAFLm3ndoD4jnJjC1T5
YnwS4FYLv24pyOLISB7XkvRv6F6lvhxstZYjDxeQws68dy/+cfDybMDS6bTgZTq1
TcNAncMegLKhWAxJOeLejy1iEwJV2EJBAPp9uFIFDxQmrZ6WC7EtCO7p2XtgVdeJ
qtJg0Ea2gasSUCa7fsPTsRP9+weHB+/fH+x7k/399wenp1387rHhw+49BDudex2w
7kdn3dqwS2IccT0Aam4ICzIyu2ZRibavgHDA5UZvgDgteRQhfItA9SAYHbduyMWV
U5r3GoLMgcSRPwnFGSzLKVirNkMojOar3lEs6kCVg7cOtY6cZw3gfuFJXdLUVfCf
Zxxzkk2d10IL1/su/eKygTBYecM65Br0Dr7CC7QoANU+gfUNKoxIF3KioQdoBO08
5WoG7+zk8Hjy6hS04uHkw/EZ+41ZjS8mpwdn/zo5GDCBPY39i4Iwq+KtJwibdrLM
/Rs46Q08C1CjNmxMFFwu3LWv3L+Ko6IU57PeH5VNW4et2hI0aFWzBNnQAEj6ygak
hkYRbaTs3G/iAV2S1lt5EeQ40qKuxhlwLIsvXfRvsjwNnAOIjVqK6GpOCAPpAils
E3FEtdzoI1q5MTQg8MMV230eDwxlmNlRhgHvZWGQmONVoU+32ujxgcsGClmlurSv
vPpcKOw29r6CpgSCPnlHA39ZV9HoekRxiDEh8GHA1HGkS2X0TQocogtL8WOVKdWo
swJB0a3zCxih4CB9PuougoTVo0sS8WKN6JgY94WS9BhM5aslBhK7BEDePcpwzotC
mBr2mKkGXPIV2OgxG5JBmCWqh4cKm4V+6XuoTgcSm1RrOmUGlmAYDBZGxZVGxPU2
Ioa8guqiCMkZEIqUIkTCKUQKuvRzapJMU9g9WvG+vKaP6JY3waACGtKLBCYPwSkj
doMqiGLFaPDVsE3cldsLpljFIomhIhiutVFK6jFo8w1++9daWQJ42yW4XYdHxwfQ
grc4eMPHzC4FeTCw2W4x0mEQgKBzGVEsQ2Nccj+EE6EAcEkSUfbgDwWMQSnFpRcY
twQQhpcqdqVMk5VwXGHbOq7FQRf19TMr3YKRiz4+zx94H/oOVPQ0Tr/aoSOeLTpp
FDOBnDfeipobbyt4MT4aZUnwxm6aZoYwcCTz846KG6eZm/4WJB6m+bVfunejNa96
GSQSxiU9gJu0yzo6Tu8sP1nbGA6L5Y5NTgPhsGv8CqgYsNWNjQGGO6i2TqFRSD7e
wIHckyJBJacyNlQH5L2HLT4GruA10SGAwECU4Bp+G+4Mt7debA/1Te0LP9xlB2K0
l6g+QBPBwYETepH4QJQYvjUN4rSgccbfwQ1M6vi/ixHACXnjIG9q3vKvXb1J1YuG
airMMHFnIEJG51jIQHUxzyhsZ6dKy1j8EfU/xBtxbDAyRUHsVRLNdJhk2ta5XzbE
VY7qUF4c62lpkwWQ2Iz4IYSLNKHc9Y3hiHYd+S9umvXe25uO6Oy1NdMCIjF9PZQV
Ky2lrWpEnV6CEFSpQh31MEjUaQ+DBC35II4Ydf4QaK1rHwStFezDSLaUMCBs7QBH
zQgr4h71nmEeCDa+cxu3NtQ22tbtoeJlSHaUOkqZzQ/TRYkiVL0y7VbLUmk0K1vV
6iMGWdMemNZAUUAk9ElRL68vi4tGAtVX8ysrKl9C7UOS+gneSvP8OgL9hoY/mUHQ
QykaEvkKh/ZceymUJqVAdVlYgVqSqkTIj7OPK1ioIM1zfZtrNsIx32IHvv9Ybw03
9bHWWSN9so+JnDvOtL3lK2xjxyZkcwhNf5AEIkWCN5ItOiR3pA54HY9bNc1H7h5d
b+prrwunBEOVhZgU5YUoxBBuz8C+L11V3atUMIIKibjuvUBHnVLV3U4En87AcsMa
PbC2yCJaKzDJTLvuQhquUycRKt0/QV/ZU3c5rYsUpIqjJRV2VqcgBbFkTUiq0Uew
bbJwALyfeH4OjYcQGPdoAKyn+GcOweZpCZzPu53HzDLQWKHxBAIUikWEZ/Gk+JyI
mwWazDHP0r7KQgMic7dtUanv2BcmEh0f2Xj/3hQCji5e/5ObXAM3kKZ0y9zFs0sI
571f4ugcekUZC7LDasVmvMWwx3HdOzUTwiPr5Wgnc5FbknnUNyLB3MHejhBswUpG
8CTSOLWa1Upk2zQOZYK5UoCBikpf+zoIdIvccrhgZWMoDSMFCxHACTOgnWo2GyH0
dM1is0hurrXgEF1GVCxif2Qd/wZiKv885h1w+T68nfw0OTqevDg+6PTM1XbbKVIx
5JKcS/ExrXcIEeb98Xy60Zao5GUkHeYc4y/rLONPK6LSP+VpdgO23P9K80DMRhpJ
35fYVLZ0oYcb1uHQwMUB2ZnCAw8sFGJwfyTyCBXtPSGIUMPV7D563xSJkBp9Utzr
aosrJcnIHqXJvTeiAMe4TrLSDkkj+PuvzWEY8JUrYZwbMNEk6O+JZKCQWTfmqJML
DEXW3WFSkUiC2hPq1eH+FL9FrczIVfE0YU274kCgzq0xwAhrtDsre9/duXgaCwfq
yOAK+X6IhUeyJKdh7UiQVgboDIj58QJak0TFIf9elleZ/lfwUcAfiSgyRV6tZSKA
MX6xsixK1ptLDidJSJU4xDrDgHG7gXSk04y2MFJzlkO7JR7XQCygGSXZShQR0XIE
Nbgsj9o3KvJrBGZeq2VHdzYZH0XtYhPxsKLt2vZaAxKhIgpukQ0REbJWl7TZjgsA
m6E3Qur7l3BYDoFiTFNa1GLhfAxH3Oip1dG2rHeTmt/RsnXrrTAtAy4426h7Z4l8
6sjDm81utSjCFD6I35Ytxylc3Yoj2MqzsR5fkYa6VAdadmWYAqCUVjYPPLyHDWwj
DqbyDba9peuUrq7Vp+H+vsf+9n1bjXe9MI91dpsPKh6GT8MvbAl89umUMjFdi0DV
rGswEHzkgncN+PPnjD3rNWNt3IU12lmAtXkX1saWxhLcebjL8ELehQfWXbhI+3T0
JbySEKn6Gsp+qnU/Qp1heipmeGEF42JCPJ+DFcE7FRBU+RDYLaZfwnpELsTOqqEW
lYPPBkKuVoG9Y6WTQr5Qbam6mpcAJKtiEF4Xw6hzRY0L0mZ4xWHbcGHFma53kNVR
gmtKFfpUTKAuajWmw8paRlE2iKPWeKKn0TcvjhLu8STE0hurZqpIZ3lg16b+jKV9
zY/NCNCxLly9aHpkRgPZGfFvPPREDzlG1u+7DVE2sII+kxy00Rdanp8bF2Ejyyxg
zrPYh6E6n/PPyY8dvD/ogPvLdK2NSuxZWYnP+bJKYfy8skK1msmySV10MyJYgCYK
VKUtWjKzuLqqC0FUgGCN2M/GlTmHy1aar62GA1SbBjpEVJ+v7j8+050XAJmS0m5X
cUaX7zmMoVJpqYrNBthxor2jjTInBV0CfYcVsSTStSODeyWpJueWyD5ccLRm8V6K
NWhjcacwWXVJqgCI/NYoyWala66rQnj/Cf7vPBHq7oGMkeRWwLgYIkWBvlDqNzz6
sviZOzd+u/cmTBn0yuMvi2siKyWXelHgTcks0JKZfsnMry8y2JJFyZIdPorjXPMI
tEPnBK8mbBXjfseateN6v4ttdlIQJ2W44sqZ46elQrlWdl26RK6c4LuRLV9UeXMV
K4shlvqtnACzGCH7ajGqulI/dQj6zKNnFTxPXadLA9o/56CkeV88ZoPjY02EsJz8
Gw9mpYorZKhO9wokr/jsX8Ez3aBmcCt926oYUTubkzCU6SJdm6tUtawetLJDDd5C
RXTIm5QF07KINBUPYK2FUU61qLKnJxYi7z4VuXpDRIpXLMoqnzVw+NRb5dlF8dzh
jybBpfAfWU/59FXrCto0aUQwvMR6EvLBiB3ohWNGVFc8iji+MU0qyiY6ijbjNlZT
tp+KL3BWFVxPcxrHfrTXlFwFaaxs2AHIAfpygsyDb5YsC21mMbC+Xw9OwMqE4ZOC
PaYCTBC8JzL3iq6fSaqZ6a0C2B9ZBzMQOGuH7bKOOHYdm7i2Uy9aj/Mk3x9U7Hmd
1ao8nSrLpjLLSt3hdXZXwaFVfgqIb97tY2Wgd/jh7cs2VgJpOJMcNqbGNi5qA6hm
CJ38ebf+xgH3kQmD5L3LyugaFCttm1In77ksEsN3Y1AFCP4litikbiivSS/AF9Z0
O2XRQ3ryuNpWAxs1gNETSGJoPPRKTL0jYIgQzq2uyTNQpazZJ/mLWC6OishTnvzr
7PW7t97k5Mj76eD96dG7tygE95xNsisEYs45naIpeI2Gf4tPrlOy6TwIsnh3rAnw
f5BdlM//9vta/uiP9f4f/EZNHlz+wXPc/f6fzdHO9si8/2c0wvf/PN3Z+P/3//wV
n6Mk5N92mTh3xTqcnlkGlrwo23u//9N+//JU1tysBzdFnqblekavGVJfONN6kQfr
dQIGN21QkHnEbzAuzOGLygNHa5ub7TCaTtlqwFZz/Mksqvv9fsNiWhuYkmUbIHZs
ONzdeLo7GrJVlL4Wjbe6utqE9czBGm3vwlyEhdPYH5p2NBoNRqMdRg04omh4yuDv
1TZjWVpE38T/wtqsBa3WY9nc/fD26GMPn2YH5UUJJSzKBKuZpOL/hSgC5gYdEETx
ipzTfwgKsIc5D/Poih3PkvByuWBYj5rzi1kMUs+/qarLor1iGUPrZWA4hHoWCqtI
wcSS5aR6qgw8gaIMMYMMuvJrmmOtZRvLJM8uscqxKNU7NwT1BcMCZUCiCseoEFdC
WMSIFd/gXGDhKz55PoP55jRQyKf+LC7XGPtQzACKyiXnqpAy5OCDIAaaRSIPq1TD
OfgrEfJw3pbyLV5uta5W9RcLd2X2BZK9sbYx2rBEm34zTTKKWGWg1o4R0NHG7taz
3W0poC0xGEphFec7hXp7OBhtb1tCjQ3PtpVQk2mnSCbIuS/3NsHcYoy3frkoY4bt
5/SyBUSwc+JvJq+OXop0DXTdghTCPxlEIXPw7Q6Ccu8yTa8K8Zqv2isfBviKDyqj
mWWWHMOEMNqv+J/lr92AtyZcFXLc7IdO1K0+IBBSkzeN7c0301ECRiKmNJahAckG
V7pnBlVz33Gvr50qgSYu8xWa9WQa0eIuYNGFvF6w4wzJ8e17eQkoe26t+c00Ot6y
mHFqXpdh7xj4Wzf12pZP7yBQEtjOAnpyWkovfsfgx3DUqChzqEaw3q9wU2+ypxw3
LnGvwuUGUVBd3ykN5FTycFdLhS7COMFHe7qq8VYxwKnPMJtS40zlLSWKN6ZgdSFr
buH03QrdTS+/aNIEz4bPBs+UeSPQNkXnh/JNcnDYHrHmt8PJPKAIVgZMVCSoJCBl
93p4/iWyfCEfqlZ6pAMzqUltEDPFAOZtqUSY6sb8lFequfoN43sQ6/7Ck/9t71h7
27aBn71fwbhYkIfaWbJsSwk6YEvc1WgbF3a6oRsGw41dJFn8QJwHgiL/fbzjkTxK
lPNojPSDCCSIZYniHY/3vgvNrj5s2G4Z+ISC8D3UJ0JthWp+ACyIrNaTi4V27VC3
lL86h28FRo8+tA8O2/vIM5O6RF3c1DzzaVHHUbFqNPL7nxGnXvKMJY4bsSZPst+g
kGp6OSfpIn9RDFT7hSTIX0eY+sJwYxti6P5OmGNAhX6QUgA+1jlCo3z1IydPXxjJ
saynmY3jbuqHsmcdaacl4UpiK29XDlegAFk5dN5dbLSCpFm3u1iZGKFcALwEdaKr
rwlQ5a4CSBBACcDLX6GXDoOJ+QxhXp5mg/SvXrGjtiCRi0rrdguedFE5dN93aV78
JbGkltSqa2kYB2EttPqabXhELH9b2PRBFE+siyqqY/kAzzZGDRxxw3wqWdXMfsNF
N78+B0UChWn2XujQKi+DN/w0EFOlAxqBj2FXPilKM3zgvr21xFxJQF8fLS3gbT5A
RmfQHIsv9VaBkZHZpNVCfrG7gH/m/4LtBAEGrNsD8Qv1YZfnC6r/o8W57q08GIHt
AqA8zWyBWqmFaxgkPlWp7qeQ6o6Ik3+rPHdXVUZ6MOjWKHS38ZQpUOp+RzmkADDe
4dFSnTQieBwSiKpmW9x9+X6d7Cm1MNYhwdWUOR2Z4FEWGVbjziqiHmPkNkeir4tI
1Kas5FBODWtovhcnX6E/8tvf/mwP+oe/HfpewRpLwGw2Du4Nm+o9I2UA+Ab+wjxU
jRz1+Po6VOQNL6gmbp1uU7QLwbe1/qDT3+/0NuibV5L+JpjWYraY+i/rdJTxZH5x
g4qEvLAlr2xhwauukaKiSvQ4KC8L2R+SpS5UkE77N3RIYCzhnZ4sJuIa+whCV6Uv
Z3hMoThfZfadXLwy0/TGk9mVqsHA84yZGsezsxG2ZgDb1SFAaPUrBkfD6QDnM9PQ
vNS19Gg2GdvcPHE2hPY7xBcqnDPsj88KGBxjCDmWQHSLBKGNLkh5RIj/m86uxTUs
E2CeCfm26zHlE2LLN3QFSRy/nGKQWJ4OCb9cOvA5zdVwqS8gyeGrj9ESibGeiUtY
nDkqJtl06VHKwoqgepj2bbGgY/rTADbGJ/D0Jd5szyPRlE4seV0gts6hcoEYsr5c
bLzbFzi9TazkYnNk+UKObRWximxfK4uvRSHDZiBVweAkJR/BtvkjdI8VzpCN63Z2
VAtYZNgUW+v5eLGZ20PC4pLtAz0Fo5vnzAxxVI8nVVYCZbVprcUy37y+lH2x+zqf
VsJSE3w8ulAjUStybYEMLdv1+3bQPVW6vZ+B89YYdf0xJJAgA0TOvEHeW+Ox2yTf
MTmzgWXiZPIJVTSPZWqL+QxTopjtqkRKgOx94+pkqI1F01h9E+fwqsRpGqRhyPwW
hQbyT0LkbWQpss98VOO001R2Mmj20JoK1H+Ahqzm+UAq73JuNIOo+xr1IJcX6FFs
aQXvwb6JcD1vv6NFZ+wOyqGS308wH0Ve0rUe7DYp5qcg53ud/l63zyZ2xbPQLBp9
vLWoIa2GesI8Fs+BNMfdMKeD/EyY9Jx3fVgzTOQxWPcRbhomQRrVmeu9FsO2NCJj
zBHECyl2RwMOMvW/xOsqk93QgyOJNVJdj67BNXtOm6dUWK43ON9IQRixc0crBS/M
LQlzwmEOa60gDKMWs74rVc1PTGhqiJElbMthND6sJqwqw9pVBViowQkueDo/O97m
LFPWj1nGvEamEm+5TKapK5KXpvVvWyidF+Whrep7mWKp1+lXfx7T7FoTS+ZRB+zl
7a4fDnjmZRZ0SOlYBiHsrv0PCdZPQGlJ8K06KLadofFN5gvBkbnoInAvxab1IE0T
TrF1oNiY+YvQQ6diGvwA6kbKqq0/Ol0upycXN6RtX/HOsmqNOoTma6V5tan8Rcws
hY8KUDfv/arA+xbWasBgwhqHBuJ9YYt7v1j84Gg2v6ETZt8LqsCVgI/Dk+kCPZJ6
4frYVTI+J7JnvCxJ23uuZ5yUm4pVb5iup0TPchVLR4VUIiFN4rguHPJCBQez2Bzj
y641YyurVFBSk8S1NP9MX1n1dZFAheWpv17TnZVvVfhfSPKn8+Hj4G23++5Weyw4
p3awote+bqbL+jO0dkdSddcY1RYXBBI0gcpZxwQHnThyJkyGR1IqzxbHuNm/KBeu
JKz+fDweXc53xHgo9UPDvaEvnsBiQOxCNkL9zk+ZcRwHYdyoaTUOimrvDBSIHrT2
ax/stcVe99PBoaJCq8wwU2FN5MMtioq0JuCoLqB7oEdZKyNOOuAlFqa7V5U3P2yC
OGuG5jj9IHBwTSujZt0DMu+ONaJ6IH9xzthMmkHYqjFZbg28CbERcpGhY7pyBJ3h
NNHv5NyTCCoQuCJew0Iyp9kykHsKIEecoj8GMlLUrMZb6HekVMyais15NZE80bzR
KPN3sikcfpR746TQMDdT5AzzzMnHm+gb653UcpTstx2m1LmV3xx3QujoYXXfuo+O
5Y+UZzrdWFWQbQDNqbSRn0ebVb8uGIIyGKYJi+nlyAUlK9UCrDm0YayCwn/II6yr
0jUN1Fc5CwBIOAobrSAKm/UlJLx8TcxF8JSL8yMwbQQS36FFINnqFU+80BMwNP2k
WbSQr5OFQblSYDWtovihVmDuG0Ek3Ldiifuk5ipWTwyO/GUk4urh8m2b3DEJp9HC
tMqVc9oJxhOM10xhZY2YA5cFNo+/KDBK4UdSI924o60YcVUdtTFJXW5MWnN1xFUu
OBBmk9bcTXI27xEA+XckqckdSd0dcTyrD4nDP+BYeakvS3x5A/eRMXFfwNlUcRAD
jJoSFVHL3evVooJHyFePkHw8/g60eCkmihKJprqVXbp6lKNEWLva8eRilqSyYM3y
87jjiLO1qQWEdDeK8ukH+njHNQlLHNoclGeDZVlqyj0h8u5WI5Uct1lj5zvTfdPh
XcyJtOb4w/OkZWExZ+22SHVHXLdAxrW4jFvJShgiC9fjxVQrasr11UOr1UdpTV5K
Q+uVVP+I6QKi6JBvujf4/VPn/WHnQOqh5m9VZiNMkc3l+XjX8+zHz4M3ve7fbXjW
/H3/Z/e6+9Ju6nc/9fbaagZ+xTvPdm4ebX8w+7tgBWjP0WfUmXe9+eejs1fHz5mB
Du8vzEEPEycFPUwEW/RWNgcdrlaipngz/kJp6OFOHO5IvqfT0OV8uSx0fOqBeeiS
xiJzPD+++2Ow3+m19w67vc9oeRjCwk+GVDCqkNl2cOR8GB6J2fTsRlnSt7v6DGQU
7m9ktwaRVfi/691B9uXyFk1Uyxfy3IVC5ShHOcpRjnKUoxzlKEc5ylGOcpSjHOUo
RznKUY5ylKMc5ShHOcpRjnKU44cd/wOYRuFDAKAAAA==
--7584767-3623878656-3248300694=:9002--


From just@letterror.com  Sun Dec  8 01:00:35 2002
From: just@letterror.com (Just van Rossum)
Date: Sun,  8 Dec 2002 02:00:35 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <r01050400-1022-44A4925E0A4611D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <r01050400-1022-77EFF7A90A4811D7AD06003065D5E7E4@[10.0.0.23]>

--7641475-0-3248301639=:9003
Content-Type: text/plain; Charset=US-ASCII
Content-Transfer-Encoding: 7bit

Just van Rossum wrote:

> - zip file names PYTHONPATH work

Doop, I forgot to include a site.py patch: unpatched it filters out
non-directories... (site.py's sys.path filtering needs to be changed more
thoroughly than this, but the attached patch is enough for zip files on
PYTHONPATH to work.)

Just
--7641475-0-3248301639=:9003
Content-Type: application/octet-stream; Name="site.py.patch"; X-Mac-Type="00000000"; X-Mac-Creator="00000000"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; Filename="site.py.patch"

SW5kZXg6IExpYi9zaXRlLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KUkNTIGZpbGU6
IC9jdnNyb290L3B5dGhvbi9weXRob24vZGlzdC9zcmMvTGliL3NpdGUucHksdgpy
ZXRyaWV2aW5nIHJldmlzaW9uIDEuNDYKZGlmZiAtYyAtcjEuNDYgc2l0ZS5weQoq
KiogTGliL3NpdGUucHkJOSBOb3YgMjAwMiAwNTowODowNiAtMDAwMAkxLjQ2Ci0t
LSBMaWIvc2l0ZS5weQk4IERlYyAyMDAyIDAwOjU3OjIzIC0wMDAwCioqKioqKioq
KioqKioqKgoqKiogNzUsODggKioqKgogIF9kaXJzX2luX3N5c19wYXRoID0ge30K
ICBmb3IgZGlyIGluIHN5cy5wYXRoOgogICAgICAjIEZpbHRlciBvdXQgcGF0aHMg
dGhhdCBkb24ndCBleGlzdCwgYnV0IGxlYXZlIGluIHRoZSBlbXB0eSBzdHJpbmcK
ISAgICAgIyBzaW5jZSBpdCdzIGEgc3BlY2lhbCBjYXNlLiBXZSBhbHNvIG5lZWQg
dG8gc3BlY2lhbC1jYXNlIHRoZSBNYWMsCiEgICAgICMgYXMgZmlsZSBuYW1lcyBh
cmUgYWxsb3dlZCBvbiBzeXMucGF0aCB0aGVyZS4KISAgICAgaWYgc3lzLnBsYXRm
b3JtICE9ICdtYWMnOgohICAgICAgICAgaWYgZGlyIGFuZCBub3Qgb3MucGF0aC5p
c2RpcihkaXIpOgohICAgICAgICAgICAgIGNvbnRpbnVlCiEgICAgIGVsc2U6CiEg
ICAgICAgICBpZiBkaXIgYW5kIG5vdCBvcy5wYXRoLmV4aXN0cyhkaXIpOgohICAg
ICAgICAgICAgIGNvbnRpbnVlCiAgICAgIGRpciwgZGlyY2FzZSA9IG1ha2VwYXRo
KGRpcikKICAgICAgaWYgbm90IGRpcmNhc2UgaW4gX2RpcnNfaW5fc3lzX3BhdGg6
CiAgICAgICAgICBMLmFwcGVuZChkaXIpCi0tLSA3NSw4NSAtLS0tCiAgX2RpcnNf
aW5fc3lzX3BhdGggPSB7fQogIGZvciBkaXIgaW4gc3lzLnBhdGg6CiAgICAgICMg
RmlsdGVyIG91dCBwYXRocyB0aGF0IGRvbid0IGV4aXN0LCBidXQgbGVhdmUgaW4g
dGhlIGVtcHR5IHN0cmluZwohICAgICAjIHNpbmNlIGl0J3MgYSBzcGVjaWFsIGNh
c2UuIEZpbGVuYW1lcyBhcmUgYWxsb3dlZCBvbiBzeXMucGF0aC4KISAgICAgIyBY
WFggd2l0aCBpbXBvcnQgaG9va3MgaXQncyBkdWJpb3VzIHRoZSBldmVuIGZpbHRl
ciBvdXQgcGF0aCBpdGVtcwohICAgICAjIHRoYXQgZG9uJ3QgZXhpc3QgaW4gdGhl
IGZpbGUgc3lzdGVtLgohICAgICBpZiBkaXIgYW5kIG5vdCBvcy5wYXRoLmV4aXN0
cyhkaXIpOgohICAgICAgICAgY29udGludWUKICAgICAgZGlyLCBkaXJjYXNlID0g
bWFrZXBhdGgoZGlyKQogICAgICBpZiBub3QgZGlyY2FzZSBpbiBfZGlyc19pbl9z
eXNfcGF0aDoKICAgICAgICAgIEwuYXBwZW5kKGRpcikK
--7641475-0-3248301639=:9003--


From barry@python.org  Sun Dec  8 02:33:23 2002
From: barry@python.org (Barry A. Warsaw)
Date: Sat, 7 Dec 2002 21:33:23 -0500
Subject: [Python-Dev] Import compromise
References: <20021207202128.GA28924@panix.com>
 <n2m-g.n0nh344e.fsf@morpheus.demon.co.uk>
 <Pine.SOL.4.50.0212071404310.3581-100000@death.OCF.Berkeley.EDU>
 <02f901c29e4b$5561be00$ced241d5@hagrid>
Message-ID: <15858.44915.895749.878026@gargle.gargle.HOWL>

>>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes:

    FL> maybe you could team up with Ping and turn his zest tool
    FL> into a brettbot?

    FL> http://www.sims.berkeley.edu/~ping/vertex-2002/zest.html

Zest rocks.  Has anything more happened with it in the last 8 months
or so?

-Barry


From brett@python.org  Sun Dec  8 03:43:11 2002
From: brett@python.org (Brett Cannon)
Date: Sat, 7 Dec 2002 19:43:11 -0800 (PST)
Subject: [Python-Dev] dummy_thread (#622537), part 2
Message-ID: <Pine.SOL.4.50.0212071938240.3581-100000@death.OCF.Berkeley.EDU>

Wrote ``dummy_threading``.  Thanks to the recent importing discussion and
one of Martin's replies to my previous announcement for this patch I
figured out how to do it while still using ``threading``.
``sys.modules`` is your friend (as long as you treat it nicely).

So the patch is ready for review again.  Tim mentioned he was going to
have a look, but I will let him say whether he is going to still look at
it now.

All relevent info is in the patch.

-Brett


From skip@manatee.mojam.com  Sun Dec  8 13:00:17 2002
From: skip@manatee.mojam.com (Skip Montanaro)
Date: Sun, 8 Dec 2002 07:00:17 -0600
Subject: [Python-Dev] Weekly Python Bug/Patch Summary
Message-ID: <200212081300.gB8D0HfZ011168@manatee.mojam.com>

Bug/Patch Summary
-----------------

316 open / 3105 total bugs (+2)
94 open / 1822 total patches (-2)

New Bugs
--------

ext module generation problem (2002-08-23)
	http://python.org/sf/599248
Tkinter: BitmapImage vanishes if not stored in non-local var (2002-11-01)
	http://python.org/sf/632323
test_poll fails on FreeBSD (2002-12-01)
	http://python.org/sf/646547
netrc & special chars in passwords (2002-12-01)
	http://python.org/sf/646592
Can't run script from IDLE (2002-12-01)
	http://python.org/sf/646730
more email ASCII decoding errors (2002-12-03)
	http://python.org/sf/648119
xmlrpc can't do proxied HTTP (2002-12-04)
	http://python.org/sf/648658
nturl2path.url2pathname() mishandles /// (2002-12-07)
	http://python.org/sf/649961
All URL funcs mishandle file://localhost (2002-12-07)
	http://python.org/sf/649962
urllib.url2pathname('file://host/')... (2002-12-07)
	http://python.org/sf/649974

New Patches
-----------

asyncore loop_once and remove poll2 (2002-12-04)
	http://python.org/sf/648322
tuple arg for issubclass (2002-12-06)
	http://python.org/sf/649608
urllib2.Request's headers are case-sens. (2002-12-06)
	http://python.org/sf/649742
Fix: asynchat.py: endless loop (2002-12-06)
	http://python.org/sf/649762
Complementary patch for OpenVMS (2002-12-07)
	http://python.org/sf/649997
posixfy some things (2002-12-08)
	http://python.org/sf/650412
avoid redefinition of pthread functions (2002-12-08)
	http://python.org/sf/650415

Closed Bugs
-----------

Removing _tkinter considered harmful (2002-03-25)
	http://python.org/sf/534748
LINKCC incorrectly set (2002-06-16)
	http://python.org/sf/569668
list slice ass ignores subtypes of list (2002-11-04)
	http://python.org/sf/633152
--with-dl-dld section of the readme (2002-11-10)
	http://python.org/sf/636313
__all__ not in RefMan index (2002-11-24)
	http://python.org/sf/643227
typo in Object/abstract.c : ternary_op (2002-11-24)
	http://python.org/sf/643260
old UnicodeData.txt (2002-12-01)
	http://python.org/sf/646408

Closed Patches
--------------

Start of a bgen tutorial (2001-09-04)
	http://python.org/sf/458383
use readline in pydoc.help if available (2002-04-28)
	http://python.org/sf/549901
Regression exceptions for cygwin (2002-05-03)
	http://python.org/sf/551977
PyBufferObject fixes (2002-05-04)
	http://python.org/sf/552438
db4 include not found (2002-08-02)
	http://python.org/sf/590377
OpenVMS patches (2002-09-24)
	http://python.org/sf/614055
make some type attrs writable (2002-11-09)
	http://python.org/sf/635933
Allow device name in linuxaudiodev.open (2002-11-27)
	http://python.org/sf/644977
Give some clue why modules fail... (2002-11-28)
	http://python.org/sf/645383


From ping@zesty.ca  Sun Dec  8 18:21:25 2002
From: ping@zesty.ca (Ka-Ping Yee)
Date: Sun, 8 Dec 2002 12:21:25 -0600 (CST)
Subject: [Python-Dev] Import compromise
In-Reply-To: <15858.44915.895749.878026@gargle.gargle.HOWL>
Message-ID: <Pine.LNX.4.33.0212081218410.9196-100000@server1.lfw.org>

On Sat, 7 Dec 2002, Barry A. Warsaw wrote:
> Zest rocks.  Has anything more happened with it in the last 8 months
> or so?

I brought a more recent revision to CSCW 2002 and demonstrated it
there.  It's not what i would call production code, though.  The
most significant recent change is that little speech-bubble icons
now appear in the message body at the ends of paragraphs that were
subsequently quoted and replied to, so you can navigate forwards
from messages to their replies.

I'm sorry, i wish it were moving along faster too.  Maybe after
the next clump of deadlines passes (CHI demo Dec 16, CHI short
Jan 3, CHI long Jan 10, Usenix Security Jan 27)...


-- ?!ng



From lists@morpheus.demon.co.uk  Sun Dec  8 17:34:15 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Sun, 08 Dec 2002 17:34:15 +0000
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <r01050400-1022-44A4925E0A4611D7AD06003065D5E7E4@[10.0.0.23]> (Just
 van Rossum's message of "Sun,  8 Dec 2002 01:44:47 +0100")
References: <r01050400-1022-44A4925E0A4611D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <fzt8l8iw.fsf@morpheus.demon.co.uk>

Just van Rossum <just@letterror.com> writes:

> I'll write a bit about the mechanics tomorrow, but in the meantime I would like
> to see some test results/bug reports/bug fixes from Windows developers. Please
> don't start with the full monty (ie. the std lib), try a small test archive
> first please ;-)

The patch applies fine to my copy of Python (downloaded from CVS Dec
1st - I can't get cvs update to cooperate for me :-() with an offset
of -3 in most of the import.c changes (I assume that's just because
I'm a bit out of date).

To get zipimport as a builtin, apply the following patch, and add
zipimport.c to the pythoncore project.

--- PC\config.c.orig	Sun Jul 07 03:59:34 2002
+++ PC\config.c	Sun Dec 08 15:37:03 2002
@@ -43,6 +43,7 @@
 extern void init_weakref(void);
 extern void init_hotshot(void);
 extern void initxxsubtype(void);
+extern void initzipimport(void);
 
 /* XXX tim: what's the purpose of ADDMODULE MARKER? */
 /* -- ADDMODULE MARKER 1 -- */
@@ -96,6 +97,7 @@
 	{"_hotshot", init_hotshot},
 
 	{"xxsubtype", initxxsubtype},
+	{"zipimport", initzipimport},
 
 /* XXX tim: what's the purpose of ADDMODULE MARKER? */
 /* -- ADDMODULE MARKER 2 -- */

Zipimport doesn't work at all for me as a dynamic extension. Is this
just a temporary problem, an issue specific to the zipimport module,
or a limitation in the hook mechanism? (I assume not the last, as that
would make it pretty useless as a general mechanism...)

As a builtin, it works fine in a release build. But a debug build
asserts in fopen() as the filename is "".

An easy fix for this is to add a check in zipimporter_init(),

    if (lenpath == 0) {
        PyErr_SetString(PyExc_ImportError, "archive path is empty");
        return -1;
    }

Do we need to set an error string there? It's not really an error, as
such.

BTW, I don't see any code for checking timestamps of .pyc files
against .py files. Did you omit that (and if so, was it deliberate or
accidental)?

I hit an odd case when I was testing interactively. I set sys.path to
a list containing just 'ziptest.zip', and then tried to import a
module. Zipimport failed because zlib wasn't accessible. That's OK,
but when I reset sys.path, zipimport *still* claimed it couldn't
decompress as zlib wasn't available. I see why - the optimisation in
get_decompress_func() only checks for zlib once - but nevertheless it
is annoying that I can only "fix" my mistake by restarting the
interpreter. This is never going to happen "in real life", so it's not
crucial. But maybe reload(zipimport) could be made to retry the check?

I also tried a Python import hook. It seemed to be doing the right
things. I didn't try too hard - just dummy {find,load}_module methods,
and an __init__. It looked like errors were generated as a result of
me not implementing the protocol right, which is good :-)

Paul.

PS The code *still* looks messy. I think that's just C though - I've
   got so used to writing Python code that *any* C looks messy :-)

-- 
This signature intentionally left blank



From just@letterror.com  Sun Dec  8 20:24:50 2002
From: just@letterror.com (Just van Rossum)
Date: Sun,  8 Dec 2002 21:24:50 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <fzt8l8iw.fsf@morpheus.demon.co.uk>
Message-ID: <r01050400-1022-1FF212970AEB11D7AD06003065D5E7E4@[10.0.0.23]>

Paul Moore wrote:

> To get zipimport as a builtin, apply the following patch, and add
> zipimport.c to the pythoncore project.

Ah, thanks, will add that to the next version.

> Zipimport doesn't work at all for me as a dynamic extension. Is this
> just a temporary problem, an issue specific to the zipimport module,
> or a limitation in the hook mechanism? (I assume not the last, as that
> would make it pretty useless as a general mechanism...)

I don't know, as you don't give specifics about what error you get, if any.
Maybe -v tells us more. (Although zipimport.c can sure use better -v reporting.)

> As a builtin, it works fine in a release build. But a debug build
> asserts in fopen() as the filename is "".
> 
> An easy fix for this is to add a check in zipimporter_init(),

Thanks, done.

> Do we need to set an error string there? It's not really an error, as
> such.

I'm not sure. I'd like to make it easy to write hooks, and it makes most sense
to use classes as import_hooks, and since an __init__ can't return anything, an
exception is the best you can get. It's _possible_ to write a __new__ method
that returns None when it can't handle the path, but that raises the bar
somewhat for writing hooks. Lots of stuff in import.c raises ImportError when it
really means: "I can't find what you're asking for". (Btw. I intend to add a
ZipImportError exception to the zipimport module, which would be a subclass of
ImportError.)

> BTW, I don't see any code for checking timestamps of .pyc files
> against .py files. Did you omit that (and if so, was it deliberate or
> accidental)?

I copied that verbatim from the PEP 273 patch ;-) I'm sure this could be fixed,
but there are more important issues at the moment.

> I hit an odd case when I was testing interactively. I set sys.path to
> a list containing just 'ziptest.zip', and then tried to import a
> module. Zipimport failed because zlib wasn't accessible. That's OK,
> but when I reset sys.path, zipimport *still* claimed it couldn't
> decompress as zlib wasn't available. I see why - the optimisation in
> get_decompress_func() only checks for zlib once - but nevertheless it
> is annoying that I can only "fix" my mistake by restarting the
> interpreter. This is never going to happen "in real life", so it's not
> crucial. But maybe reload(zipimport) could be made to retry the check?

Right, but maybe I simply shouldn't remember that a previous zlib import failed,
and just try again.

> I also tried a Python import hook. It seemed to be doing the right
> things. I didn't try too hard - just dummy {find,load}_module methods,
> and an __init__. It looked like errors were generated as a result of
> me not implementing the protocol right, which is good :-)
> 
> Paul.
> 
> PS The code *still* looks messy. I think that's just C though - I've
>    got so used to writing Python code that *any* C looks messy :-)

import.c, zipimport.c or both? ;-) Yeah, it needs work, but you're also right
about C: there's a lot of string manipulation going on which is a huge pain in
the butt. But: I admit not very _good_ at doing string manipulations in C, so
once it's more or less bugfree I'm sure more experienced people can improve it
if needed.

Just


From lists@morpheus.demon.co.uk  Sun Dec  8 22:38:37 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Sun, 08 Dec 2002 22:38:37 +0000
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <r01050400-1022-1FF212970AEB11D7AD06003065D5E7E4@[10.0.0.23]> (Just
 van Rossum's message of "Sun,  8 Dec 2002 21:24:50 +0100")
References: <fzt8l8iw.fsf@morpheus.demon.co.uk>
 <r01050400-1022-1FF212970AEB11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <y970jfv6.fsf@morpheus.demon.co.uk>

Just van Rossum <just@letterror.com> writes:

> Paul Moore wrote:

>> Zipimport doesn't work at all for me as a dynamic extension. Is
>> this just a temporary problem, an issue specific to the zipimport
>> module, or a limitation in the hook mechanism? (I assume not the
>> last, as that would make it pretty useless as a general
>> mechanism...)
>
> I don't know, as you don't give specifics about what error you get,
> if any.  Maybe -v tells us more. (Although zipimport.c can sure use
> better -v reporting.)

Sorry. Basically it just didn't find the module in the zipfile. I
don't think it's a big deal. If you want, I can rebuild as a dynamic
module and try with -v.

>> Do we need to set an error string there? It's not really an error,
>> as such.
>
> I'm not sure. I'd like to make it easy to write hooks, and it makes
> most sense to use classes as import_hooks, and since an __init__
> can't return anything, an exception is the best you can get.

Ah. Now I see the logic (actually, it's the same as iu.py). I'd say
stick with the exception method. What I was really thinking was in the
C code, could I just return -1 without setting the exception. But
that's not good practice, even if it did work.

> It's _possible_ to write a __new__ method that returns None when it
> can't handle the path, but that raises the bar somewhat for writing
> hooks. 

Too right. Leave it as it is.

>> BTW, I don't see any code for checking timestamps of .pyc files
>> against .py files. Did you omit that (and if so, was it deliberate
>> or accidental)?
>
> I copied that verbatim from the PEP 273 patch ;-) I'm sure this
> could be fixed, but there are more important issues at the moment.

It's a fair cop guvnor :-)

To be honest, I think that this is one of the nastiest hurdles to
writing a good import hook - replicating (or even understanding the
logic for!) the hairy extension and timestamp checking code used for
normal imports. Refactoring and encapsulating that would potentially
help in a lot of cases. Except that maybe it's not always exactly the
same (you can't write a compiled .pyc file back to a zip file, you
can't load a C extension from a zipfile, ...) But that isn't an issue
for this patch.

>> I hit an odd case when I was testing interactively. I set sys.path to
>> a list containing just 'ziptest.zip', and then tried to import a
>> module. Zipimport failed because zlib wasn't accessible. That's OK,
>> but when I reset sys.path, zipimport *still* claimed it couldn't
>> decompress as zlib wasn't available. I see why - the optimisation in
>> get_decompress_func() only checks for zlib once - but nevertheless it
>> is annoying that I can only "fix" my mistake by restarting the
>> interpreter. This is never going to happen "in real life", so it's not
>> crucial. But maybe reload(zipimport) could be made to retry the check?
>
> Right, but maybe I simply shouldn't remember that a previous zlib
> import failed, and just try again.

I thought of that - it slows things down when zlib isn't available, as
it checks (and fails) every time. How bad this is, I'm not sure...

>> PS The code *still* looks messy. I think that's just C though - I've
>>    got so used to writing Python code that *any* C looks messy :-)
>
> import.c, zipimport.c or both? ;-)

import.c hasn't got much better - but at least it hasn't got any worse
:-)

> Yeah, it needs work, but you're also right about C: there's a lot of
> string manipulation going on which is a huge pain in the butt. But:
> I admit not very _good_ at doing string manipulations in C, so once
> it's more or less bugfree I'm sure more experienced people can
> improve it if needed.

It wasn't a criticism. Every time I look at C code which manipulates
strings (you're right, that's the bulk of the nastiness) it reminds me
how *nice* virtually any other language is.

But you let me write import hooks in Python and avoid C, so you're my
friend for life :-)

Paul.
-- 
This signature intentionally left blank



From fredrik@pythonware.com  Sun Dec  8 22:53:28 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Sun, 8 Dec 2002 23:53:28 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
References: <fzt8l8iw.fsf@morpheus.demon.co.uk><r01050400-1022-1FF212970AEB11D7AD06003065D5E7E4@[10.0.0.23]> <y970jfv6.fsf@morpheus.demon.co.uk>
Message-ID: <000b01c29f0c$a192eda0$ced241d5@hagrid>

paul wrote:

> To be honest, I think that this is one of the nastiest hurdles to
> writing a good import hook - replicating (or even understanding the
> logic for!) the hairy extension and timestamp checking code used for
> normal imports. Refactoring and encapsulating that would potentially
> help in a lot of cases. Except that maybe it's not always exactly the
> same (you can't write a compiled .pyc file back to a zip file, you
> can't load a C extension from a zipfile, ...) But that isn't an issue
> for this patch.

do you have real use cases for this?  I would have thought that the
vast majority of users of a feature like this would use it to "freeze" a
collection of modules for use with their application.

emulating a file system when you already have a file system seems a
bit like overkill...

</F>



From just@letterror.com  Sun Dec  8 22:59:37 2002
From: just@letterror.com (Just van Rossum)
Date: Sun,  8 Dec 2002 23:59:37 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <y970jfv6.fsf@morpheus.demon.co.uk>
Message-ID: <r01050400-1022-BE62AAD80B0011D7AD06003065D5E7E4@[10.0.0.23]>

Paul Moore wrote:

> Sorry. Basically it just didn't find the module in the zipfile. I
> don't think it's a big deal. If you want, I can rebuild as a dynamic
> module and try with -v.

Don't worry, I've found (and solved) the problem and am playing with a dynamic
build as we speak.

[time stamp checking]
> To be honest, I think that this is one of the nastiest hurdles to
> writing a good import hook - replicating (or even understanding the
> logic for!) the hairy extension and timestamp checking code used for
> normal imports.

Not all import hooks will need to support all of .pyc, .pyc and .py. I for one
would've been more than happy for zipimport to only support .pyc ;-)

> Refactoring and encapsulating that would potentially
> help in a lot of cases. Except that maybe it's not always exactly the
> same (you can't write a compiled .pyc file back to a zip file, you
> can't load a C extension from a zipfile, ...) But that isn't an issue
> for this patch.

Right; I need to peek in the zipfile.py module to see how the time/date shorts
from the zip archive translate to a usable timestamp. I've updated my code so it
at least doesn't throw away those values ;-)

> > Right, but maybe I simply shouldn't remember that a previous zlib
> > import failed, and just try again.
> 
> I thought of that - it slows things down when zlib isn't available, as
> it checks (and fails) every time. How bad this is, I'm not sure...

That was my original rationale, too, but it's wrong: it only checks if
compression is *needed* and *if* zlib is not available then, the program will
abort with a traceback anyway... I've changed it in my local copy.

[ ... ]
> But you let me write import hooks in Python and avoid C, so you're my
> friend for life :-)

Heh. Make sure to buy me a beer some time -- and I'll be sure to buy you one
right back ;-)

Just


From just@letterror.com  Sun Dec  8 23:04:16 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  9 Dec 2002 00:04:16 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <000b01c29f0c$a192eda0$ced241d5@hagrid>
Message-ID: <r01050400-1022-6314D23B0B0111D7AD06003065D5E7E4@[10.0.0.23]>

Fredrik Lundh wrote:

> emulating a file system when you already have a file system seems a
> bit like overkill...

True, but zipimport (just like the PEP 273 implementation) *does* handle any of
.py, .pyc and .pyo. It prefers byte code over source, but ignores the time
stamp. That *could* be confusing.

Just


From gward@python.net  Sun Dec  8 23:09:41 2002
From: gward@python.net (Greg Ward)
Date: Sun, 8 Dec 2002 18:09:41 -0500
Subject: [Python-Dev] __getstate__() not inherited when __slots__ present
Message-ID: <20021208230941.GA12533@cthulhu.gerg.ca>

If a class and its superclass both define __slots__, it appears that
__getstate__() is not inherited from the superclass.  Example:

"""
class Base (object):
    __slots__ = []

    def __getstate__ (self):
        return tuple([getattr(self, attr) for attr in self.__slots__])

class Thing (Base):
    __slots__ = ['a', 'b']

    def __init__ (self):
        self.a = 42
        self.b = "boo!"

abstract = Base()
thing = Thing()

print abstract.__getstate__
print thing.__getstate__
"""

When I run this with a not-quite-current CVS Python:

  <bound method Base.__getstate__ of <__main__.Base object at 0x401f73b8>>
  <built-in function __getstate__>

The upshot of this is that I can't just define one __getstate__() in the
superclass of a bunch of __slots__-using classes -- I guess I'll have to
set __getstate__ manually for each class.  ;-(  If this is a feature, is
it documented anywhere?

(BTW, I see the same behaviour with Python 2.2.2.)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
"Very funny, Scotty.  Now beam my *clothes* down."


From skip@pobox.com  Sun Dec  8 23:53:24 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sun, 8 Dec 2002 17:53:24 -0600
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <r01050400-1022-6314D23B0B0111D7AD06003065D5E7E4@[10.0.0.23]>
References: <000b01c29f0c$a192eda0$ced241d5@hagrid>
 <r01050400-1022-6314D23B0B0111D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <15859.56180.914141.702803@montanaro.dyndns.org>

    Just> It prefers byte code over source, but ignores the time stamp. That
    Just> *could* be confusing.

Not to mention a change of semantics.  If the .py file is newer than the
.pyc or .pyo files, why not just force compilation but don't save the result
(maybe also toss a Warning)?  Isn't that what importing from a read-only
directory or filesystem does today?  Sure, it would slow things down, but it
would better match the semantics of importing from the filesystem.
Presumably testing before delivery of the zip archive would identify any
time order problems.

Skip


From barry@python.org  Mon Dec  9 04:28:24 2002
From: barry@python.org (Barry A. Warsaw)
Date: Sun, 8 Dec 2002 23:28:24 -0500
Subject: [Python-Dev] Import compromise
References: <15858.44915.895749.878026@gargle.gargle.HOWL>
 <Pine.LNX.4.33.0212081218410.9196-100000@server1.lfw.org>
Message-ID: <15860.7144.998931.300423@gargle.gargle.HOWL>

>>>>> "KY" == Ka-Ping Yee <ping@zesty.ca> writes:

    KY> I'm sorry, i wish it were moving along faster too.  Maybe
    KY> after the next clump of deadlines passes (CHI demo Dec 16, CHI
    KY> short Jan 3, CHI long Jan 10, Usenix Security Jan 27)...

I can sympathize!  Zest is one of those Cool Projects to keep an eye
on.

-Barry


From just@letterror.com  Mon Dec  9 08:54:45 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  9 Dec 2002 09:54:45 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <15859.56180.914141.702803@montanaro.dyndns.org>
Message-ID: <r01050400-1022-E02087810B5311D7AD06003065D5E7E4@[10.0.0.23]>

Skip Montanaro wrote:

>     Just> It prefers byte code over source, but ignores the time stamp. That
>     Just> *could* be confusing.
> 
> Not to mention a change of semantics.

Hm, questionable, as we don't currently *have* zip import ;-)

> If the .py file is newer than the
> ..pyc or .pyo files, why not just force compilation but don't save the result
> (maybe also toss a Warning)?  Isn't that what importing from a read-only
> directory or filesystem does today? 

Yes, this is what it should do. However, I'd be *very* grateful for a recipe
that translates a dos time/date (it's two shorts, zipfile.py contains code that
translates it to a time tuple) to something compatible with st.st_mtime. And it
can't use the time module ;-)

> Sure, it would slow things down, but it
> would better match the semantics of importing from the filesystem.
> Presumably testing before delivery of the zip archive would identify any
> time order problems.

Agreed.

Just


From mal@lemburg.com  Mon Dec  9 10:15:38 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 09 Dec 2002 11:15:38 +0100
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
References: <20021206025220.E17658@vestdata.no>
Message-ID: <3DF46D4A.2070902@lemburg.com>

Ragnar Kj=F8rstad wrote:
> Hi
>=20
> The dumps-method in xmlrpclib has the following comment:
>     All 8-bit strings in the data structure are assumed to use the
>     packet encoding.  Unicode strings are automatically converted,
>     where necessary.
>=20
>=20
> This doesn't work very well. In our particular case we're using latin_1
> as our default encoding, and we're using UTF-8 for the packet encoding.
> We can't really change the default encoding, because the sql-modules
> transfer latin_1 encoded data and we can't change the packet encoding t=
o
> latin_1 because the xmlrpc-client (php) doesn't work with that.
 >
> The attached patch changes xmlrpclib to convert strings to unicode usin=
g
> the default encoding, and then convert them back to strings with the
> packet encoding. If unicode is not available it falls back to the old
> behaviour.

I believe this is overkill. If you need this behaviour, subclass
the Marshaller in xmlrpclib and add your feature to that subclass.
Then replace the Marshaller class in xmlrpclib with your subclass.

Aside: xmlrpclib should support subclassing the Marshaller and
Unmarshaller more transparently. Currently, the two are hard-coded
into the rest of xmlrpclib without the possibility to provide your
own subclasses without tweaking xmlrpclib from the outside.

> I guess for performance it could check if the defaultencoding is the
> same as the packet-encoding, but my guess is that it hardly ever is, so
> no reason to optimize for it.
>=20
> Note; I'm not at all sure this is the best way to fix the problem. If
> it's not, please feel free to ignore this patch, or even better - tell
> me what the preferable fix is.

Please post patches using the SourceForge patch manager.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From mchermside@ingdirect.com  Mon Dec  9 13:33:02 2002
From: mchermside@ingdirect.com (Chermside, Michael)
Date: Mon, 9 Dec 2002 08:33:02 -0500
Subject: [Python-Dev] Re: Another approach for the import mechanism
Message-ID: <902A1E710FEAB740966EC991C3A38A8903C2791A@INGDEXCHANGEC1.ingdirect.com>

Gustavo Niemeyer writes:
> - Don't have to change path to use compressed packages (at least
>   not if you want to provide compressed packages, individual
>   compressed modules or the standard library).
>=20
> - Don't have to specify the compression type hardcoded.
>=20
> - Allows one to ship a package inside a zip file, without asking
>   the user to change his path, and without hacking the package.
>=20
> - Allows one to compress a single file (foobar.py.bz2).
>=20
> I belive that my propose is quite clear now. If there are no
> additional supporters, there's no reason to go on.

I think this proposal would make sense IF compression were an
important goal here. But to me, it isn't. Zip does two things...
it aggregates into a single file (maintaining directory structure)
and it compresses. Of the two, I find the aggregation important
and the compression a mere side effect.

This is why I really don't care much about switching to a different
compression (or aggregation) format -- one standard way to do it
is more useful to me than a BETTER way. It is why I don't care about
compressing a single file. Basically, disk space is cheap, but
effort to keep track of (and distribute) complex file hierarchies
isn't.

> Thanks to everyone who discussed.

And thanks for your contributions too. I don't realize exactly
what I was looking for from .zip until you explained so clearly
the benefits of your proposal.

-- Michael Chermside



From bkline@rksystems.com  Mon Dec  9 14:33:56 2002
From: bkline@rksystems.com (Bob Kline)
Date: Mon, 9 Dec 2002 09:33:56 -0500 (EST)
Subject: [Python-Dev] Re: Another approach for the import mechanism
In-Reply-To: <902A1E710FEAB740966EC991C3A38A8903C2791A@INGDEXCHANGEC1.ingdirect.com>
Message-ID: <Pine.LNX.4.33.0212090925380.15698-100000@rksystems.com>

On Mon, 9 Dec 2002, Chermside, Michael wrote:

> I think this proposal would make sense IF compression were an
> important goal here. But to me, it isn't. Zip does two things... it
> aggregates into a single file (maintaining directory structure) and
> it compresses. Of the two, I find the aggregation important and the
> compression a mere side effect.

Our *three* chief weapons are ....  Oops, wrong show.

Zip (and tar + bzip2/gzip) provide at least three services:
  1. aggregation
  2. compression
  3. integrity validation (CRC, etc.)

I'm leaving off encryption, so someone else can continue with the SI
gag.

Some do a better job than others at letting you get to the rest of the
archive if one spot is corrupted.

> Basically, disk space is cheap ...

True in many contexts, but not all.  And the differences in compression
ratios between zip and bzipped tar can be surprisingly significant.

-- 
Bob Kline
mailto:bkline@rksystems.com
http://www.rksystems.com




From fredrik@pythonware.com  Mon Dec  9 14:29:12 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 9 Dec 2002 15:29:12 +0100
Subject: [Python-Dev] Re: Another approach for the import mechanism
References: <Pine.LNX.4.33.0212090925380.15698-100000@rksystems.com>
Message-ID: <012b01c29f8f$58ed6d90$0900a8c0@spiff>

Bob Kline wrote:

> > Basically, disk space is cheap ...
>=20
> True in many contexts, but not all.  And the differences in =
compression
> ratios between zip and bzipped tar can be surprisingly significant.

tried doing random access in a bzipped tar file?

</F>



From skip@pobox.com  Mon Dec  9 14:36:01 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 9 Dec 2002 08:36:01 -0600
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <r01050400-1022-E02087810B5311D7AD06003065D5E7E4@[10.0.0.23]>
References: <15859.56180.914141.702803@montanaro.dyndns.org>
 <r01050400-1022-E02087810B5311D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <15860.43601.620525.747883@montanaro.dyndns.org>

    Just> However, I'd be *very* grateful for a recipe that translates a dos
    Just> time/date (it's two shorts, zipfile.py contains code that
    Just> translates it to a time tuple) to something compatible with
    Just> st.st_mtime. And it can't use the time module ;-)

Looking at the code in zipfile.py which converts a time tuple to DOS date
and time shorts (I didn't see code that went the other way), it appears that
dosdate is 7 bits for the year, 4 bits for the month and 5 bits for the day.
Dostime looks like 5 bits for the hour, 6 bits for the minute and 5 bits for
half the number of seconds.  You should be able to extract those chunks into
a tm structure then call the C lib mktime() function to convert to a time in
seconds since the epoch (this is just off the top of my head):

    stm = malloc(sizeof(struct tm));
    stm->tm_sec = (dostime & 0x1f) * 2;
    stm->tm_min = (dostime >> 5) & 0x3f;
    stm->tm_hour = dostime >> 11);
    stm->tm_mday = dosdate & 0x1f;
    stm->tm_mon = (dosdate >> 5) & 0xf;
    stm->tm_year = dosdate >> 9;
    mtime = mktime(stm);

Here's the layout of dosdate and dostime as I understand it from reading the
zipfile code:

    dosdate = yyyyyyymmmmddddd
    dostime = hhhhhmmmmmmsssss

where "sssss" is actually the seconds divided by 2.  I'll let others check
my code.  I have to dash off to a class.

Skip


From just@letterror.com  Mon Dec  9 14:35:50 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  9 Dec 2002 15:35:50 +0100
Subject: [Python-Dev] Re: Another approach for the import mechanism
In-Reply-To: <Pine.LNX.4.33.0212090925380.15698-100000@rksystems.com>
Message-ID: <r01050400-1022-86A4F4B10B8311D7AD06003065D5E7E4@[10.0.0.23]>

Bob Kline wrote:

>   3. integrity validation (CRC, etc.)

This is another thing that zipimport doesn't do (nor does the PEP 273
implementation btw.). It could in theory be added but it doesn't mean much:
someone can still put bogus data in a zip file and make marshal crash. So I'm
tempted to blame marshal and forget about CRC checking...

Just


From just@letterror.com  Mon Dec  9 14:56:03 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  9 Dec 2002 15:56:03 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <15860.43601.620525.747883@montanaro.dyndns.org>
Message-ID: <r01050400-1022-594A64B80B8611D7AD06003065D5E7E4@[10.0.0.23]>

Skip Montanaro wrote:

> Looking at the code in zipfile.py which converts a time tuple to DOS date
> and time shorts (I didn't see code that went the other way), it appears that
> dosdate is 7 bits for the year, 4 bits for the month and 5 bits for the day.
> Dostime looks like 5 bits for the hour, 6 bits for the minute and 5 bits for
> half the number of seconds.  You should be able to extract those chunks into
> a tm structure then call the C lib mktime() function to convert to a time in
> seconds since the epoch (this is just off the top of my head):
> 
>     stm = malloc(sizeof(struct tm));
>     stm->tm_sec = (dostime & 0x1f) * 2;
>     stm->tm_min = (dostime >> 5) & 0x3f;
>     stm->tm_hour = dostime >> 11);
>     stm->tm_mday = dosdate & 0x1f;
>     stm->tm_mon = (dosdate >> 5) & 0xf;
>     stm->tm_year = dosdate >> 9;
>     mtime = mktime(stm);
> 
> Here's the layout of dosdate and dostime as I understand it from reading the
> zipfile code:
> 
>     dosdate = yyyyyyymmmmddddd
>     dostime = hhhhhmmmmmmsssss
> 
> where "sssss" is actually the seconds divided by 2.  I'll let others check
> my code.  I have to dash off to a class.

Thanks! How portable is this? Which header(s) should I include?

Just


From guido@python.org  Mon Dec  9 15:13:45 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Dec 2002 10:13:45 -0500
Subject: [Python-Dev] __getstate__() not inherited when __slots__ present
In-Reply-To: Your message of "Sun, 08 Dec 2002 18:09:41 EST."
 <20021208230941.GA12533@cthulhu.gerg.ca>
References: <20021208230941.GA12533@cthulhu.gerg.ca>
Message-ID: <200212091513.gB9FDjb19635@odiug.zope.com>

> If a class and its superclass both define __slots__, it appears that
> __getstate__() is not inherited from the superclass.  Example:
> 
> """
> class Base (object):
>     __slots__ = []
> 
>     def __getstate__ (self):
>         return tuple([getattr(self, attr) for attr in self.__slots__])
> 
> class Thing (Base):
>     __slots__ = ['a', 'b']
> 
>     def __init__ (self):
>         self.a = 42
>         self.b = "boo!"
> 
> abstract = Base()
> thing = Thing()
> 
> print abstract.__getstate__
> print thing.__getstate__
> """
> 
> When I run this with a not-quite-current CVS Python:
> 
>   <bound method Base.__getstate__ of <__main__.Base object at 0x401f73b8>>
>   <built-in function __getstate__>
> 
> The upshot of this is that I can't just define one __getstate__() in the
> superclass of a bunch of __slots__-using classes -- I guess I'll have to
> set __getstate__ manually for each class.  ;-(  If this is a feature, is
> it documented anywhere?
> 
> (BTW, I see the same behaviour with Python 2.2.2.)

It's a feature.  When a class defines __slots__ and not __getstate__,
it gets a bogus __getstate__ that always raises an exception.  The
assumption is that the base class __getstate__ doesn't know about the
subclass slots and hence is unlikely to be able to retrieve them
correctly.

BTW, your code for accessing the slots by saying self.__slots__
doesn't work in general: self.__slots__ returns the __slots__ variable
of the most derived class only.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jacobs@penguin.theopalgroup.com  Mon Dec  9 15:20:09 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Mon, 9 Dec 2002 10:20:09 -0500 (EST)
Subject: [Python-Dev] __getstate__() not inherited when __slots__ present
In-Reply-To: <200212091513.gB9FDjb19635@odiug.zope.com>
Message-ID: <Pine.LNX.4.44.0212091016200.7957-100000@penguin.theopalgroup.com>

On Mon, 9 Dec 2002, Guido van Rossum wrote:
> It's a feature.  When a class defines __slots__ and not __getstate__,
> it gets a bogus __getstate__ that always raises an exception.  The
> assumption is that the base class __getstate__ doesn't know about the
> subclass slots and hence is unlikely to be able to retrieve them
> correctly.
> 
> BTW, your code for accessing the slots by saying self.__slots__
> doesn't work in general: self.__slots__ returns the __slots__ variable
> of the most derived class only.

Here is my code for dealing with this "feature":

  def __getstate__(self):
    state = getattr(self, '__dict__', {}).copy()
    for obj in type(self).mro():
      for name in getattr(obj,'__slots__',()):
        if hasattr(self, name):
          state[name] = getattr(self, name)
    return state

  def __setstate__(self, state):
    for key,value in state.items():
      setattr(self, key, value)

I also have a meta-class that adds these methods for derived objects, since
the core decided to play it overly safe on this issue.

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From guido@python.org  Mon Dec  9 15:29:37 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Dec 2002 10:29:37 -0500
Subject: [Python-Dev] Proposed changes to linuxaudiodev
In-Reply-To: Your message of "Sat, 07 Dec 2002 16:23:26 EST."
 <20021207212326.GA13707@panix.com>
References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca>
 <20021207212326.GA13707@panix.com>
Message-ID: <200212091529.gB9FTbG19825@odiug.zope.com>

> > Sounds fine to me.  Should I worry about adding an official
> > DeprecationWarning to linuxaudiodev.c?  Or does that wait until 2.4?
> 
> Given its lack of official documentation, I'd vote for DeprecationWarning
> now.

+1.

Somebody check this in please.  (But make sure that running the unit
tests doesn't issue the warning.)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec  9 15:30:33 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Dec 2002 10:30:33 -0500
Subject: [Python-Dev] Import compromise
In-Reply-To: Your message of "Sat, 07 Dec 2002 16:11:38 EST."
 <20021207211138.GB10103@panix.com>
References: <20021207202128.GA28924@panix.com> <m3ptsdim8e.fsf@mira.informatik.hu-berlin.de>
 <20021207211138.GB10103@panix.com>
Message-ID: <200212091530.gB9FUXA19837@odiug.zope.com>

[Aahz]
> You may be right; however, the tone of the discussion seems to be partly
> predicated on a short timeframe.  I'm fairly confident Guido won't hold
> 2.3 for lack of a new import facility.

No, I'll push the zipfile import developers harder to be ready before
Xmas. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec  9 15:34:38 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Dec 2002 10:34:38 -0500
Subject: [Python-Dev] Zip imports, PEP 273, and the .zip extension
In-Reply-To: Your message of "Sat, 07 Dec 2002 15:56:00 GMT."
 <n2m-g.bs3xq0vj.fsf@morpheus.demon.co.uk>
References: <n2m-g.bs3xq0vj.fsf@morpheus.demon.co.uk>
Message-ID: <200212091534.gB9FYcL19877@odiug.zope.com>

> PEP 273 does not mandate a particular file extension for zip
> files. However, both current implementations (Jim Ahlstrom's patch and
> Just's import hooks) recognise zip files via the extension ".zip".
> 
> I'd like to suggest that this should not be the case, and that
> zipfiles should be recognised by the embedded signature, as is done by
> zipfile.is_zipfile(). This will (slightly) slow down the initial
> checking for zipfiles, as it requires a file read, but if the result
> of the check is cached the performance impact will be minimal.
> 
> The main advantage of this is in generality. However, I have a
> specific usage in mind - namely, concatenating a Python script with
> its supporting code as a zip archive. (Zip archives can have arbitrary
> data prepended without affecting their validity). Then, simply by
> executing
> 
>     sys.path.append(sys.argv[0])
> 
> the script can access its own embedded modules.
> 
> One problem with getting this to work is that at present, the Python
> parser will choke on data appended to the script. This can be avoided
> under Windows by a system-specific hack, of embedding a ^Z character
> in the script (which text-mode I/O treats as EOF) just before the
> appended data. In the longer term, a language-mandated "stop parsing
> here" token, like Perl's __END__, might be useful. But that's not
> something I'm proposing right now (unless an expert in Python's parser
> steps up and says "oh, that's easy - I can implement that in a couple
> of lines of code" :-))
> 
> Any comments?

-1.  Too much of a hack.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Mon Dec  9 15:37:37 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 9 Dec 2002 16:37:37 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
References: <r01050400-1022-594A64B80B8611D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <001b01c29f98$f08a3530$ced241d5@hagrid>

just wrote:

> > Looking at the code in zipfile.py which converts a time tuple to DOS date
> > and time shorts (I didn't see code that went the other way), it appears that
> > dosdate is 7 bits for the year, 4 bits for the month and 5 bits for the day.
> > Dostime looks like 5 bits for the hour, 6 bits for the minute and 5 bits for
> > half the number of seconds.  You should be able to extract those chunks into
> > a tm structure then call the C lib mktime() function to convert to a time in
> > seconds since the epoch (this is just off the top of my head):
> > 
> >     stm = malloc(sizeof(struct tm));
> >     stm->tm_sec = (dostime & 0x1f) * 2;
> >     stm->tm_min = (dostime >> 5) & 0x3f;
> >     stm->tm_hour = dostime >> 11);
> >     stm->tm_mday = dosdate & 0x1f;
> >     stm->tm_mon = (dosdate >> 5) & 0xf;
> >     stm->tm_year = dosdate >> 9;

I could have sworn zip files stored UTC time, but the specification
only says "time", and zipfile.py uses localtime.  oh well...  here's a
slightly more robust (but still untested) version:

        #include <time.h>

        struct tm stm;
        time_t mtime;

        stm.tm_sec = (dostime & 0x1f) * 2;
        stm.tm_min = (dostime >> 5) & 0x3f;
        stm.tm_hour = (dostime >> 11) & 0x1f;
        stm.tm_mday = dosdate & 0x1f;
        stm.tm_mon = ((dosdate >> 5) & 0xf) - 1;
        stm.tm_year = ((dosdate >> 9) & 0x7f) + 80;
        stm.tm_isdst = 0; /* wday/yday is ignored */

        mtime = mktime(&stm);

you may have to add 1 to the mday (mktime expects 1..31) and/or
remove the -1 from the mon value (mktime expects 0..11).

> Thanks! How portable is this? Which header(s) should I include?

mktime is part of ANSI C.  for the header, see above.

</F>



From pedronis@bluewin.ch  Mon Dec  9 15:41:54 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Mon, 9 Dec 2002 16:41:54 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
References: <r01050400-1022-E02087810B5311D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <023401c29f99$80358900$6d94fea9@newmexico>

I haven't looked at the code. Is that true that this version requires all
import hooks to be triggered by some kind of string in sys.path
which is then recognized by the import hook.?

Finn has already said that but I repeat it, there are cases for Jython for
which this cannot work, involving import hooks that correspond
to runtime only objects.

regards.



From guido@python.org  Mon Dec  9 16:02:48 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Dec 2002 11:02:48 -0500
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: Your message of "Mon, 09 Dec 2002 16:41:54 +0100."
 <023401c29f99$80358900$6d94fea9@newmexico>
References: <r01050400-1022-E02087810B5311D7AD06003065D5E7E4@[10.0.0.23]>
 <023401c29f99$80358900$6d94fea9@newmexico>
Message-ID: <200212091602.gB9G2ms20175@odiug.zope.com>

> Finn has already said that but I repeat it, there are cases for Jython for
> which this cannot work, involving import hooks that correspond
> to runtime only objects.

Can you give an example?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pedronis@bluewin.ch  Mon Dec  9 16:33:33 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Mon, 9 Dec 2002 17:33:33 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
References: <r01050400-1022-E02087810B5311D7AD06003065D5E7E4@[10.0.0.23]>              <023401c29f99$80358900$6d94fea9@newmexico>  <200212091602.gB9G2ms20175@odiug.zope.com>
Message-ID: <025201c29fa0$b753c760$6d94fea9@newmexico>

From: "Guido van Rossum" <guido@python.org>


> > Finn has already said that but I repeat it, there are cases for Jython for
> > which this cannot work, involving import hooks that correspond
> > to runtime only objects.
>
> Can you give an example?


Java class loaders,

http://java.sun.com/j2se/1.4.1/docs/api/java/lang/ClassLoader.html

from them one can retrieve java classes or data resources.

E.g. when Jython is embedded, the embedding enviroment can have setup
classloaders to access Jython code and other bundled data
( getResource, getResourceAsStream).
And our users would like to bundle and access Python code from there.

They are purely runtime object, and an abstraction over the filesystem, zip
files or some remote code repository of whatever...

regards.



From jim@interet.com  Mon Dec  9 17:24:11 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 09 Dec 2002 12:24:11 -0500
Subject: [Python-Dev] Complexity of import.c
Message-ID: <3DF4D1BB.9030400@interet.com>

I see that my patch to import.c has been criticized
as adding to the complexity of an already complex module.
I don't entirely agree.  If you want to have the standard
library available as a zip module, you must have zip file
import available in C.  The patch I proposed in PEP 273
was a minimalist patch; it adds as little as possible
while still keeping package import and preserving all
import semantics.

IMHO, the real complexity of import.c comes not from
the double loop
   for path in sys.path:
      for suffix in (".pyc", ".pyo",...):
          fopen(...)
and not from the caching of zip archive names.  It comes
from the use of import.c to perform package imports.
Package imports are performed by creating recursive calls
to functions that would otherwise be flat, and by replacing
sys.path with the package path.  This is darned confusing.

At the time PEP 273 was discussed, I proposed moving package
imports out of import.c into a Python module such as Gordon's
iu.py or Greg's imputil.py.  This was rejected because package
users feared that package imports would be slowed down.  The
speed of Python startup and import was a concern.

I understand that people want to generalize zip imports, and
create a better import hook mechanism.  Some feel that
now is the time to do it.

This is my humble opinion:

Replacing import.c plus zip import with an equally complex
import.c is a fundamental mistake.  The current import.c
is understood (by few), and the zip file import patch adds
the minimum required.  If a different import.c is accepted,
then it should be MUCH simpler, and support only flat
directory searches and zip directories.  Package imports
and import hooks should be moved to a Python module such
as Gordon's iu.py.  And import.c must bootstrap iu.py and
all its imports from either a directory search or a zip file.

An improved import hook mechanism should have its own PEP.

I tried my best not to break the Macintosh port, which has
a lot of special code.  A replacement import.c should do
the same.

My patch modified many other Python .c and .py files to
solve several difficult bootstrap problems.  If you replace
just import.c you will get a painful lesson in bootstrapping.
You probably need these other patches even if you reject my
import.c patch.

Jim Ahlstrom



From guido@python.org  Mon Dec  9 17:39:07 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Dec 2002 12:39:07 -0500
Subject: [Python-Dev] New and Improved Import Hooks
In-Reply-To: Your message of "Sat, 07 Dec 2002 00:55:13 +0100."
 <r01050400-1022-2EF6075E097611D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-2EF6075E097611D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212091739.gB9Hd7121989@odiug.zope.com>

> Guido van Rossum wrote:
> 
> > The problem with trying to load x.y right away is that then you can't
> > distinguish between "x.y doesn't exist" (and then you go to step 2)
> > as opposed to "x.y exists, but it tries to import something else that
> > raised ImportError" (and then you should simply propagate the
> > exception).

[His brother replied]
> Also not true: a function that finds *and* loads a module can return
> None for "module not found", a new module object upon succes, and
> propagate an error if one did occur during import. I'm tempted to
> make my new import hook scheme work that way: it avoids a few hoops
> I have to go through with import.c, as right now I'm quite limited
> what zipimporter.find_module() can communicate to
> zipimporter.load_module(). (It would also make it even closer to
> iu.py ;-)

OK, sounds like it can work, if b/w compat issues are taken care of.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Mon Dec  9 17:43:18 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Mon, 9 Dec 2002 18:43:18 +0100
Subject: [Python-Dev] Complexity of import.c
References: <3DF4D1BB.9030400@interet.com>
Message-ID: <018701c29faa$75f7b560$f92fe8d9@mira>

> I see that my patch to import.c has been criticized
> as adding to the complexity of an already complex module.
> I don't entirely agree.  If you want to have the standard
> library available as a zip module, you must have zip file
> import available in C.  The patch I proposed in PEP 273
> was a minimalist patch; it adds as little as possible
> while still keeping package import and preserving all
> import semantics.

This is not the case. Much of the complexity of the patch stems from an
(atleast perceived) attempt to reorganize import.c. For example:

- why did is_builtin have to go?
- Was the ExternalNames dictionary really necessary?
- Why was the HAVE_READLINK chunk moved?
- What is the purpose of Get/SetScriptName?

Regards,
Martin



From just@letterror.com  Mon Dec  9 18:03:59 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  9 Dec 2002 19:03:59 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
In-Reply-To: <025201c29fa0$b753c760$6d94fea9@newmexico>
Message-ID: <r01050400-1022-A153A7460BA011D7AD06003065D5E7E4@[10.0.0.23]>

Samuele Pedroni wrote:

> Java class loaders,
> 
> http://java.sun.com/j2se/1.4.1/docs/api/java/lang/ClassLoader.html
> 
> from them one can retrieve java classes or data resources.
> 
> E.g. when Jython is embedded, the embedding enviroment can have setup
> classloaders to access Jython code and other bundled data
> ( getResource, getResourceAsStream).
> And our users would like to bundle and access Python code from there.
> 
> They are purely runtime object, and an abstraction over the filesystem, zip
> files or some remote code repository of whatever...

With my current patch you could just add a "special" string to sys.path, and
associate an importer object with it. However, this sounds like a job for iu's
metapath (which I also intend to implement, if only to show it's possible). Very
much simplified, this allows you to add inport hooks that *don't* correspond to
sys.path items.

I'm almost done with my note about this stuff; will explain in more detail
there.

Just


From just@letterror.com  Mon Dec  9 18:21:24 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  9 Dec 2002 19:21:24 +0100
Subject: [Python-Dev] Complexity of import.c
In-Reply-To: <3DF4D1BB.9030400@interet.com>
Message-ID: <r01050400-1022-0C5FF53E0BA311D7AD06003065D5E7E4@[10.0.0.23]>

James C. Ahlstrom wrote:

> I see that my patch to import.c has been criticized
> as adding to the complexity of an already complex module.
> I don't entirely agree.  If you want to have the standard
> library available as a zip module, you must have zip file
> import available in C.

This has not been questioned.

> The patch I proposed in PEP 273
> was a minimalist patch; it adds as little as possible
> while still keeping package import and preserving all
> import semantics.

I think my counterpatch (even if it's not done yet) shows this is simply not
true.

> IMHO, the real complexity of import.c comes not from
> the double loop
>    for path in sys.path:
>       for suffix in (".pyc", ".pyo",...):
>           fopen(...)
> and not from the caching of zip archive names.  It comes
> from the use of import.c to perform package imports.
> Package imports are performed by creating recursive calls
> to functions that would otherwise be flat, and by replacing
> sys.path with the package path. 

Erm, I think package import is very recursive by *nature*. I'm not sure what
function you mean that could be flat.

> This is darned confusing.

So true!

> At the time PEP 273 was discussed, I proposed moving package
> imports out of import.c into a Python module such as Gordon's
> iu.py or Greg's imputil.py.  This was rejected because package
> users feared that package imports would be slowed down.  The
> speed of Python startup and import was a concern.

It's also dubious how this would help: the current code works well and is
stable. I would fear that moving any of this to Python would cause bootstrapping
problems.

> I understand that people want to generalize zip imports, and
> create a better import hook mechanism.  Some feel that
> now is the time to do it.
> 
> This is my humble opinion:
> 
> Replacing import.c plus zip import with an equally complex
> import.c is a fundamental mistake.

Yes, and luckily noone is proposing to do that.

[ ... ]
> My patch modified many other Python .c and .py files to
> solve several difficult bootstrap problems.  If you replace
> just import.c you will get a painful lesson in bootstrapping.

I've learned them indeed over the last couple of days, but I didmanage to get it
working by only touching import.c and importdl.h. But my current working version
also touches pythonrun.[hc], as it has a new _PyImportHooks_Init function. (The
reason this was neccesary was I wanted a subclass of ImportError in the
zipimport module; I learned the hard way that you can't create new exceptions
from C before the exceptions module is loaded... So I had to move my init code
from _PyImport_Init to a separate function, which is called by Py_Initialize().)

> You probably need these other patches even if you reject my
> import.c patch.

I doubt it.

Just


From Wiktor Sadowski" <art@wiktorsadowski.com  Sun Dec  8 18:46:50 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Sun, 8 Dec 2002 19:46:50 +0100
Subject: [Python-Dev] zipimport, round 3 (or would that be that 37?)
Message-ID: <006001c29eea$2dfa27e0$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_005D_01C29EF2.8D2854A0
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

Just van Rossum <just@letterror.com> writes:

> I'll write a bit about the mechanics tomorrow, but in the meantime I =
would like
> to see some test results/bug reports/bug fixes from Windows =
developers. Please
> don't start with the full monty (ie. the std lib), try a small test =
archive
> first please ;-)

I have managed to apply the patch (import.c) to Python2.2.2 (wasn't too =
hard)

Could you please email me the site.py patch,or post it again?=20
>>>base64.decodestring(site.patch)
>>>binascii.Error: Incorrect padding

Python 2.2.2 (#37, Dec  7 2002, 19:19:43) [MSC 32 bit (Intel)] on win32
Type "copyright", "credits" or "license" for more information.

>>> import sys
>>> print sys.import_hooks
[<type 'zipimport.zipimporter'>]

>>> from PIL import Image
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in ?
    from PIL import Image
ImportError: No module named PIL

>>> sys.path.append('C:\Python22\PIL.zip')
>>> from PIL import Image
now it works

>>> from zipimport import zipimporter
>>> arc=3Dzipimporter('C:\Python22\Modules.zip')  #Modules.zip - zipped =
non-package directory
>>> m=3Darc.load_module('module1')
ok

>>> arc1=3Dzipimporter('C:\Python22\Modules.zip\Folder1')
>>> m=3Darc1.load_module('fmodule1')
ok


Regards
Wiktor
art@wiktorsadowski.com


------=_NextPart_000_005D_01C29EF2.8D2854A0
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV>Just van Rossum &lt;<A=20
href=3D"mailto:just@letterror.com">just@letterror.com</A>&gt; =
writes:</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>&gt; I'll write a bit about the mechanics tomorrow, but in the =
meantime I=20
would like<BR>&gt; to see some test results/bug reports/bug fixes from =
Windows=20
developers. Please<BR>&gt; don't start with the full monty (ie. the std =
lib),=20
try a small test archive<BR>&gt; first please ;-)</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>I have managed to apply the&nbsp;patch (import.c) to Python2.2.2 =
(wasn't=20
too hard)</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Could you please email me the site.py patch,or post it again?=20
<BR>&gt;&gt;&gt;base64.decodestring(site.patch)<BR>&gt;&gt;&gt;binascii.E=
rror:=20
Incorrect padding</DIV>
<DIV>&nbsp;</DIV>
<DIV>Python 2.2.2 (#37, Dec&nbsp; 7 2002, 19:19:43) [MSC 32 bit (Intel)] =
on=20
win32<BR>Type "copyright", "credits" or "license" for more=20
information.<BR><BR>&gt;&gt;&gt; import sys<BR>&gt;&gt;&gt; print=20
sys.import_hooks<BR>[&lt;type 'zipimport.zipimporter'&gt;]</DIV>
<DIV>&nbsp;</DIV>
<DIV>&gt;&gt;&gt; from PIL import Image<BR>Traceback (most recent call=20
last):<BR>&nbsp; File "&lt;pyshell#2&gt;", line 1, in =
?<BR>&nbsp;&nbsp;&nbsp;=20
from PIL import Image<BR>ImportError: No module named PIL</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>&gt;&gt;&gt; sys.path.append('C:\Python22\PIL.zip')<BR>&gt;&gt;&gt; =
from=20
PIL import Image<BR>now it works</DIV>
<DIV>&nbsp;</DIV>
<DIV>&gt;&gt;&gt; from zipimport import zipimporter<BR>&gt;&gt;&gt;=20
arc=3Dzipimporter('C:\Python22\Modules.zip')&nbsp; #Modules.zip - zipped =

non-package directory<BR>&gt;&gt;&gt; =
m=3Darc.load_module('module1')<BR>ok</DIV>
<DIV>&nbsp;</DIV>
<DIV>&gt;&gt;&gt;=20
arc1=3Dzipimporter('C:\Python22\Modules.zip\Folder1')<BR>&gt;&gt;&gt;=20
m=3Darc1.load_module('fmodule1')<BR>ok</DIV>
<DIV>&nbsp;</DIV>
<DIV><BR>Regards<BR>Wiktor<BR><A=20
href=3D"mailto:art@wiktorsadowski.com">art@wiktorsadowski.com</A><BR></DI=
V></BODY></HTML>

------=_NextPart_000_005D_01C29EF2.8D2854A0--



From jim@interet.com  Mon Dec  9 21:31:37 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 09 Dec 2002 16:31:37 -0500
Subject: [Python-Dev] Complexity of import.c
References: <3DF4D1BB.9030400@interet.com> <018701c29faa$75f7b560$f92fe8d9@mira>
Message-ID: <3DF50BB9.2080603@interet.com>

Martin v. Löwis wrote:
>>I see that my patch to import.c has been criticized
>>as adding to the complexity of an already complex module.

> This is not the case. Much of the complexity of the patch stems from an
> (atleast perceived) attempt to reorganize import.c. For example:

I think package imports really are to blame, because they require
the zip import mechanism to be within import.c.  Remember that
PEP 273 semantics require that a zip archive is equivalent to
a directory hierarchy, not just a single directory.  Then this
must work on __path__:
      /a/b/c/myarchive.zip/subdir/
And consider a package import which changes the path, and then
imports subpackages, each of which may also alter the path.
I don't see how you can jump out of import.c on a sys.path item
and make this work, because the new path is coming from package
import, not the original sys.path.

> - why did is_builtin have to go?

I introduced a dictionary of zip file contents to avoid
repeated linear searches of the zip file.  Since builtins
are a constant list, it was easy to add builtin names too.
I was not necessary to ditch is_builtin, but it was more
natural to do so.

> - Was the ExternalNames dictionary really necessary?

Yikes!  that is where I record the zip file contents.  That
avoids the linear search.

OK, so once I had a dictionary of zip names, it was natural
to add other info, such as directory file names too.

> - Why was the HAVE_READLINK chunk moved?
> - What is the purpose of Get/SetScriptName?

These changes and several others were necessary to be
able to import the Python standard lib from a zip file.
The problem was the attempt to import site.py before the
sys.path was fully set up.  This is arguably a bug in
Python.  The changes moved all sys.path setup to getpath
and out of sysmodule.c.

This isn't an issue of zip importing per se.  It is the
avoidance of phase errors during start up.  For example,
you need a full valid sys.path to be able to import zlib.pyd
(on Windows) so you read a compressed zip archive.  Then
you import site.py plus whatever it imports.  This must be
available in the zip standard lib file, so zip importing
must already work.  Thus the need for careful bootstrapping.

JimA



From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Mon Dec  9 21:44:33 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Mon, 9 Dec 2002 22:44:33 +0100
Subject: [Python-Dev] Complexity of import.c
References: <3DF4D1BB.9030400@interet.com> <018701c29faa$75f7b560$f92fe8d9@mira> <3DF50BB9.2080603@interet.com>
Message-ID: <029f01c29fcc$29aa3300$f92fe8d9@mira>

> > - Was the ExternalNames dictionary really necessary?
>
> Yikes!  that is where I record the zip file contents.  That
> avoids the linear search.

So what would have been the problem with linear search that makes this
extra data structure strictly necessary to have?

> This isn't an issue of zip importing per se.  It is the
> avoidance of phase errors during start up.  For example,
> you need a full valid sys.path to be able to import zlib.pyd
> (on Windows) so you read a compressed zip archive.  Then
> you import site.py plus whatever it imports.  This must be
> available in the zip standard lib file, so zip importing
> must already work.  Thus the need for careful bootstrapping.

I think you can't really cover all cases. For example, your patch fails
to find zlib in the Unix build tree, as the build directory is added to
sys.path only in site.py, and your patch won't attempt to import zlib
outside InitZip. So it might have been good to explain somewhere what
bootstrapping scenarios this patch is meant to support.

Regards,
Martin



From arigo@tunes.org  Mon Dec  9 22:48:15 2002
From: arigo@tunes.org (Armin Rigo)
Date: Mon,  9 Dec 2002 14:48:15 -0800 (PST)
Subject: [Python-Dev] os.path.commonprefix
In-Reply-To: <15858.8941.920830.141997@montanaro.dyndns.org>; from skip@pobox.com on Sat, Dec 07, 2002 at 10:33:49AM -0600
References: <20021207160218.1528F49BA@bespin.org> <15858.8941.920830.141997@montanaro.dyndns.org>
Message-ID: <20021209224815.8F7D94B52@bespin.org>

Hello Skip,

On Sat, Dec 07, 2002 at 10:33:49AM -0600, Skip Montanaro wrote:
> :-) I discovered the same thing over a year ago, fixed the problem in CVS,
> then got roundly booed because I'd changed the semantics.

Can't we deprecate the thing and move it elsewhere? (where?)

In all cases, wouldn't it be worth a documentation note?  I could submit a patch.


Armin



From just@letterror.com  Mon Dec  9 22:51:21 2002
From: just@letterror.com (Just van Rossum)
Date: Mon,  9 Dec 2002 23:51:21 +0100
Subject: [Python-Dev] zipimport, round 4
Message-ID: <r01050400-1022-C5B5252B0BC811D7AD06003065D5E7E4@[10.0.0.23]>

--11364638-0-3248466697=:9007
Content-Type: text/plain; Charset=US-ASCII
Content-Transfer-Encoding: 7bit

Here's a new patch. Changes include:
- pyc time stamp checking (thanks Skip and </F>!)
- better python -v output
- also works when zipimport is built dynamically

I've written a note about various aspects of the patch (pasted below) but I'm
not sure it's PEP-ready yet. Comments are more than welcome!

Just

---------------------

This note is in a way an addendum to PEP 273. I fully agree the points and goals
of the PEP, except for the sections "Efficiency", "Directory Imports" and
"Custom Imports". However, I disagree strongly with much of the design and
implementation of PEP 273. This note presents an alternative, with a matching
implementation.


A brief history of import.c -- digging through its cvs log.

When Python was brand new, there were only builtin modules and .py files. Then
.pyc support was added and about half a year later support for dynamically
loaded C extension was implemented. Then came frozen modules. Then Guido rewrote
much of import.c, introducing the filedescr struct {suffix, mode, type},
allowing for some level of (builtin) extension of the import mechanism. This was
just before Python 1.1 was released. Since then, the only big change has been
package support (in 1997), which added a lot of complexity. (The __import__ hook
was quietly added in 1995, it's not even mentioned in the log entry of ceval.c
r2.69, I had to do a manual binary search to find it...)

All later import extensions were either implemented using the filedescr
mechanism and/or hardcoded in find_module and load_module. This ranges from
reading byte code from Macintosh resources to Windows registry-based imports.
Every single time this involved another test in find_module() and another branch
in the load_module() switch. "This has to stop."


The PEP 273 implementation.

Obviously the PEP 273 implementation has to add *something* to import.c, but it
makes a few mistakes:

  - it's badly factored (for example it adds the full implementation of
    reading zip files to import.c.)
  - it adds a questionable new feature: directory caching. The original
    author claimed this was needed for zip imports, but instead solving
    the problem locally for zip files the feature is added to the builtin
    import mechanism as a whole. Although this causes some speedup
    (especially for network file system imports), this is bad, for several
    reasons:
    - it's not strictly *needed* for builtin import
    - it's not a frequent feature request from users (as far as I know)
    - it makes import.c even more complicated than it already is (note that
      I say "complicated", not "complex")
    - it changes semantics: if a module is added to the file system *after*
      the directory contents has been cached, it will not be found. This
      might only be a real issue for an IDE that runs code inside the IDE
      process, but still.


A different approach.

An __import__ hook is close to useless for the problem at hand, as it needs to
reimplement much of import.c from scratch. This can be witnessed in Guido's old
ihooks.py, Greg Stein's imputils.py and Gordon McMillan's iu.py, each of which
are failry complex, and not compatible with any other. So we either have to add
just another import type to import.c for zip archives, or we can add a more
general import hook. Let's assume for a moment we want to do the *former*.

The most important goal is for zip file names on sys.path and PYTHONPATH to
"just work" -- as if a zip archive is just another directory. So when traversing
sys.path, each item must be checked for zip-file-ness, and if it is, the zip
file's file index needs to be read so we can determine whether the module being
imported is in there.

I went for an OO approach, and represent a zip file with an instance of the
zipimporter class. Obviously it's quite expensive to read the zip file index
again and again, so we have to maintain a cache of zipimporter objects. The most
Pythonic approach would be to use a dict, using the sys.path item as the key.
This cache could be private to the zip import mechanism, but it makes sense to
also cache the fact that a sys.path item is *not* a zip file. A simple solution
is to map such a path item to None. By now it makes more sense to have this
cache available in import.c.


The zipimporter protocol.

The zipimporter's constructor takes one argument: a path to a zip archive. It
will raise an exception if the file is not found or if it's not a zip file.

The import mechanism works in two steps: 1) find the module, 2) if found, load
the module. The zipimporter object follows this pattern, it has two methods: 

    find_module(name):
        Returns None if the module wasn't found, or the
        zipimporter object itself if it was.

    load_module(fullname):
        Load the module, return it (or propagate an exception).

The main path traversing loop in import.c will then look like this (omitting the
caching mechanics for brevity):

    def find_module(name, path):
        if isbuiltin(name):
            return builtin_filedescr
        if isfrozen(name):
            return frozen_filedescr
        if path is None:
            path = sys.path
        for p in sys.path:
            try:
                v = zipimporter(p)
            except ZipImportError:
                pass
            else:
                w = v.find_module(name)
                if w is not None:
                    return w
            ...handle builtin file system import...


Packages.

Paths to subdirectories of the zip archive must also work, on sys.path for one,
but most importantly for pkg.__path__. For example: "Archive.zip/distutils/".
Such a path will most likely be added *after* "StdLib.zip" has been read (after
all, the parent package is *always* loaded before any submodules), so all I need
to do is strip the sub path, and look up the bare .zip path in the cache. A
*new* zipimporter instance is then created, which references the same (internal,
but visible) file directory info as the "bare" zipimporter object. A .prefix
contains the sub path:

  >>> from zipimport import zipimporter
  >>> z = zipimporter("Archive.zip/distutils")
  >>> z.archive
  'Archive.zip'
  >>> z.prefix
  'distutils/'
  >>> 


Beyond zipimport.

So there we are, zipimport works, with just a relatively minor impact on
import.c. The obvious next question is: what about other import types, whether
future needs for the core, or third party needs? It turns out the above approach
is *easily* generalized to handle *arbitrary* path-based import hooks. Instead
of just checking for zip-ness, it can check a list of candidates (again, caching
cruft omitted):

    def find_module(name, path):
        if isbuiltin(name):
            return builtin_filedescr
        if isfrozen(name):
            return frozen_filedescr
        if path is None:
            path = sys.path
        for p in sys.path:
            v = None
            for hook in sys.import_hooks:
                try:
                    v = hook(p)
                except ImportError:
                    pass
                else:
                    break
            if v is not None:
                w = v.find_module(name)
                if w is not None:
                    return w
            ...handle builtin file system import...

Now, one tiny step further, and we have something that fairly closely mimics
Gordon McMillan's iu.py. That tiny step is what Gordon calls the "metapath". It
works like this:

    def find_module(name, path):
        for v in sys.meta_path:
            w = v.find_module(name, path)
            if w is not None:
                return w
        # fall through to builtin behavior
        if isbuiltin(name):
            return builtin_filedescr
        [ rest same as above ]

An item on sys.meta_path can override *anything*, and does not need an item on
sys.path to get invoked. The find_module() method of such an object has an extra
argument: path. It is None or parent.__path__. If it's None, sys.path is
implied.


The Patch.

I've modified import.c to support all of the above. Even the path handler cache
is exposed: sys.path_importers. (This is what Gordon calls "shadowpath"; I'm not
happy with either name, I'm open to suggestions.) The sys.meta_path addition
isn't strictly neccesary, but it's a useful feature and I think generalizes and
exposes the import mechanism to the maximum of what is possible with the current
state of import.c.

The protocol details are open to discussion. They are partly based on what's
relatively easily doable in import.c. Other than that I've tried to follow
common sense as to what is practical for writing import hooks.

The patch is not yet complete, especially regarding integration with the imp
module: you can't currently use the imp module to invoke any import hook. I have
some ideas on how to do this, but I'd like to focus on the basics first. Also:
the reload() function is currently not supported. This will be easy to fix
later.

I *thought* about allowing objects on sys.path (which would then work as an
importer object) but for now I've not done it as sys.meta_path makes it somewhat
redundant. It would be easy to do, though: it would add another 10 lines or so
to import.c.

I've tested the zipimporter module both as a builtin and as a shared lib: it
works for me in both configurations. But when building it dynamically: it _has_
to be available on sys.path *before* site.py is run. When running from the build
dir on unix: add the appropriate build/lib.* dir to your PYTHONPATH and it
should work.
--11364638-0-3248466697=:9007
Content-Type: application/octet-stream; Name="zipper4.tgz"; X-Mac-Type="00000000"; X-Mac-Creator="00000000"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; Filename="zipper4.tgz"

H4sIAAAAAAAAA+09+VMbR7P5VforBqUCEixYEhJn7BQxEPM+GyjAuV1bi7SCjSWt
vLviSML//vqYc3fFkcTOq+9ZlVjs7ExPT09PX9Mz+j2aTMKk8+yLj/hpNjvN9W4X
vpvN9TX+brY7/M2fL6DCerfdWmu11r9otlqrnfYXovsxkVKfaZoFiRBf/AZ/3FcP
qg0GnwKhT/v5Xc4/fEejSZxkK71/vI9mq9lc63RmzH9nrd3t5OZ/tdtd/UI0/3FM
Sj7/z+f/y2jcG077oagd32aX8XjlslY1ZWmWTHvZKBydh4n7Jk774SB1y0ZBkl4G
Q7ewF48m0TB0Cr/OohGUvKhWq0DWLOqJ3mWQLIrJrR/eZOE4jeJx+ss78Vz8UVuZ
3PZqnsDvWH7X7rah5bNFoZk2TER8/lvYywRgFY2jDACIYNwX6XSC78Xis2o1u52E
8FrwoJzGP0eTA/n3NuJUqPBHtXJ8e0Rd+K/2dnbNo1gMkt5ldBVuCwEoTYLschyM
QhEPRHYZIhAhKyASVrMBUCWFRtSsH0HRRTgOk2Ao8I2IxoM412KSwOButrkFVeKS
LVELnqXT82f9KIGKcXL7rIZN77Y1fY9vz2D0EpA1Wh+Lt00t1ZWuspckMdIEe4yT
6yDpA4l7QwRfaJSEQd/XONRxTjV1GiWdDGOoDwTyR3F/OgzrFl5iMQ2HA08wjMF0
OESietUKF0ADJLQHVMpElE6C3vvgIlSPkuX6nkA+8zMxwm/PJmUW9/xwnCW3Bi9u
66fhhDGHP+Bl9UviqNAh2svLsPe+Hk8aejBER1Xsifk8iRvV4vCrFoP54/C67kzS
Ym+YesLmsgvn+f11P21UgS9hZg4uxnESAp9dTEcwKqhng8b1ILLgfZiKXpBIzgwy
oAdwEi6S61D04/FCBg9QMRYjqCuG0SAUQIc+TAbMPKyk894wSNMwXSG2LEwW0KqC
37Bq3ZlswEiWX2QTPxgO416dxtUE0laigahTi7nn4vDt69cNXGYEY/mFWjP8ZluX
06oplPI6MMV31UoSZtNkLOqGYg3G8q5qzXg1T6gyJnzEJFiLFPjSVyCh8qKWIRo7
Yq/z6eCXNzs/Hu+cvXq9d7jUeqdeLDJnL+bHhPw5DMdIZiTc3PHtTnLhH4PMDc+m
E1g+jFztaG7LGtSKT8Py/RrMdwVnHD7zx7enWRKNL4g3gV0dmjcaUFOSb7mF/R3f
+geHL0/29utuRcAKkQUcNcCdlP8o1gTc7Yqn0e9hoRIPjWo+Bx4hfji+BRnkn4aZ
BOyKJhiwYhVCJUpFOJpktzXs0h4F8oQC/uK5MKR/VC9Iu4rbUxbHYhiPL2b2VMdq
v0B/y613OJ6FyYL480+RLz1eaIj5eVqMwjRp88vIbSJLD0qbrPLL390msvRnq4l5
2eGXKwtMBM1yNMAlYjdSNkRSVJxAGlr9MMRwmIbUCt4rwoyCW3GudR+IjVGVeK72
DCE+y+JnsuYKsCiqLNAWpKoqivdl50hSlDr17W1GrUIqD4QVAl4EQZQtirjXmyZJ
MO5RjzWEKYFJaB8AGiAN/9UnnqxAs0Xz8wHHTnKHZjdf+eeD49mVz0HXvaeXH8TS
c9HR9aQGWfzQ4IqapB+oykT9dafZRNWwZSAMdhwLxBeIOIVR06iYR/eBqkFWzqDj
GIQ42RwoJ7fEVyvtZjMFswlpWsanvCgVDsu6HlBh3Jvc1kFKcWMPmQFYoqUEtxoo
N23YDLS0xFyTvgdETveOGXsUeAADrbqFX5sLiAz3Dn3BHxIHQpLkm+yAi7EloNd6
JzuqmGIECJ1su6WIqt0V01tKK0Blu1pQNFo07SfxSIoBTQ9XzZRWVfgbxWZB59nF
lemCMkzlSFyg3llyi8r4IsQpBX7uXYZ9xyYdQN8ivU1XXJWzQov8B63T+30Eg8um
hzBEPAWNMbxSetxtzGO7Tf3vwow1Wr3m1qgpBsg1lDochAxA2AVzVlpJbjU5fZZO
lJWhv4MsHOWqM1vDeArahHlQgbE6L9ppqlbDSJJ0HGS9S7K6U6ajpk9feRIsSFyr
o56zbDRoWYWYELTlj466pFcNw4W8jl8OwyCp23aQ7scWBfgSIH4fJudxGu4Pg4sG
d4Kz9EMSZeFpBjZaUq99abhjS6AtjsKa3ISaImNNW+c86oWv0oVfx7Z8cMebs+h1
rVkYYz85CSMfm7bhdRVHfcfy6odsGxaML2lc+T/u7pXbH4V3mtiFN0bA8HN87qNL
SKbpIAnZGmnkDUQ0hn0Eig7IbBeF3BP5NwlN/nNx4htHBR60r0IDM/aea98Bh/74
448e/kMGOjDnWKQxeJVQsiXGYUhruh9bZgx0CMwu9TMJcC2/HzLOjHUmRTG0wjJL
YpOArVbkcJSFQJKU7QSUtw4rSwHnmvYaKdNoNnpKsyhNsVRuPZppvavO6AEnpzBI
xczKGDJ6RJUs2RqE1+GbYDLB/l8F6X/CWwddYjy5kni49nwDoNa2IYExsBjDmjbT
7eFubruWVg5gE6va6EtUzaojfrp7yPdEu0r54KYKiMG860NMW1gCRYcPmmpvBow8
XhvSr3H9nneS8SOyLzzjyz/o5qSOl2MNwXF0ZO+2S8PulGTVLIE1kNQ1kmQKOzyb
dwzg+abnfx8Mp6FteUkIYjRNM4FmGJjBAazQLAv7K/jGcRO0m6p40pUyLFg0UrSI
55l880aCbBuH5mtwloojJAM6IkYREdRZha+lJWcpGj70ctG36J2W9E9k+5y7yDb0
THdcKUWQeTx/RD1j9Rpg8NdhPKZxS2iy5BEsTqGmf57FSwJKmvF10MpaA0XmL6iA
kpVQscNYT1wY1sCdhaGwm7Uy5rSoZSPK1Rhgx0rvs0zYoxlW3l6J9CctLWVDoyaU
HQUZtDifZmB60IKDxcbOaTprman1SeKf1ryZH1z0SnepasagMS1Vi20Wyead4/AA
NFLBf3NpP1skCit1y+NeUi3FL0vgVGul8Q7ewNqlFkUlR8WoEp9rR32J/LIlAg9q
3bIYbIlAwXZERnM3GexKGkibfaY4kDQ1jR0fgZhWcocJx+oghapAmt56DzOzxuxD
y4EQOhhnYDy8jomHuJnyJ3RLT3TYlAChIc1COwis9KwFt8mTio745BbMr2nSAya8
CqJhcD7kaP59EtbIBlh+Ooz3aNELFf8y1cvJLqWzGyNHyRZhjTbLacY1H5lnrjUr
htjWxN2Ve4EfO/zOcXfh0HnmBDDW1P9zJdklyqgczoM+LKULEPHjKW5HgUUtzm8z
8NriPvYB4xwDDaXbZnUCEBl8pRePs2g8DenpztJKUEUqooJvpt21ewKQltvVC9Dn
RvJJbVaz1BVzgK2r3oTZZdzfDQdOwH5EpXIDrFr5o+ZYN2KG6eaJN3tnr/zvd052
Tr47BbTAV6jdedjeUQJihl6c3R7x9ioV/Loj7zlEOobDwh7TG9ooLA4HS63hSCEE
qIgz/+jb/9l7eeaJeDBIwywe2HrWU/Kq4VVO9nZ2jw5f/8QjIoZHAA+DYIcwD4AF
KkB4qLlUWbn2RAsa/PHtLnDx6dkJEK7u+LRxz6siEcEK7NEWSnG7smbvMO3u7e+d
nOzt+ju7uyd7p6d1/G6I5uM28Zi4ziYlGE4HZ/UC2HmGw/tSoByaMCDDwysWlmgx
pCB/XZo0PGxTkesSnOcIxBFWo7VX74e8fxonjRIX35Nt5CM1cYBNEnKV88UXYYZK
v/ginfUCxRBudxVeJOGkpHqQ+lKwlL1Kww/TEAPNZS9HLJmL7y6D9LIEMRh5yTjk
GPQMfoe7wVEPxP0OjM/LESKeSYmSN4AjSOxBqHrwz473X+98dwoicn/n7esz8aew
Cr/dOd07++l4zxPcejAMLlJqmWdv3UG/bCazJLiCdV9Csx6K15KJiXqXM2ftOgze
D6M041VafB9lZVOHpVotlMhYMwRZUFKRpJddkQpKWbQUs/OgjAa0418sDdNegpBm
vSrtAWFZdKmjVThJ4p6zALFQcxHtCTMzkCyQzLbDS1TzjV6iua1qUwUeXLbdDYee
wQzjakpNYA4BABmGuEcdUIoGpWWwczNBRwYYKcXnOsoA+LsfZHJXX74g90zmZ2Qj
sHJHtO2cjVayEdCmh+aMrCrmRfOmNWiIRdHe1nVG0diu8+KF6Dao5urAVLoEWy9X
q9VqSIAWrH6AppnEs/g6pq7q6r3pC5BapnCQrHmLyyFXdZOrrkPdJbHRNJWjtA8e
D5mcQNBrwOHZLSISpSKiPIA+RwCVbfOeCDoPrTm2CY4FaM1cZFc7dssvWGFWsdJ3
UDQWgdrvlFtLsi5agdGwj1EOYD8w9qQUpIgzmokpgqgDBwVDtQOhm05TrIo+SJAC
hDSE6Q9QZVBNYDo0C6MwXSE8dowJicMMRIY2/ha9Iy8UpUYSpqnHz18KVYBM9h5s
pG3RJBU8Has3YV+1FkDywEcF5snWpMzigTB1qY4AYP0ofa8b4lBLG/bDXFO3Ca9V
j5rIdYuNsAve1MmChIokvXRraVCrz5doZFKSC1ZG0LpmL+mZekSRpMch4nQ6ytdu
ECV3JTdElCgyRqcaCuKLMQyqD7Y2zSAI9Wio5g7WEpZxao5NSPLVraEKFOnNlSoy
ZjFQc2/ekJtdkUv/2D94vQeFE4ztgvcnDB/Q+K2JtabKmQKWMBxYYRiXgA1IOVUB
Bycbyjf4UAjKcFsM3lD85gId+l48HWcqxkNxWitcvyS6Ov6Dfcx6tzixgpXo0uul
+uLBJAYOqByBFh4M42s7poLrmFY1BRNgTZVmM5hsGstnNUY5xRhxR3sQTwxi4Dkk
53qXEN8Xw6nle9glnhQCtvazazqCVVvgve2Fmo1PCeYwi+F7QMMTy+22h24uqqZT
KOS1FgKnyuVAikwFPCmV0T+BKce4Au7z7kMVAETx4eZNc63Z7XzbbeoN+2+D/pbY
Y2gvUVaB2IMlBTLhYhwAUgy+MugN45TgbD+FHPmd/adTAkghF4/ccj0Mr/X2Y2Eb
7yIGQRzK5D+biGsehwqcZSIDFLNpRgEtcarkmkUfuR2JtOF1g4oQObGR26eh1SR3
PZzUECsIS7EPqAWIUwTF4J1b1c0GB3c0RjT/XInnafiYMXEAXjJEu9kihsCZ4fwQ
zRY2P2Bz8crqaQaSuDHUlJlyFSXYCkidXgJ75LHSgbEHa5K18aiaqD8eQxGjAx5T
W4vlR9XWsvhxKFvyGhp01oCiBsISZyY9AOaR1bbvncZOW02jrRIfy14GZUfgI5fZ
9DCviPNRLMtYdSFKqZtZ0coKpV6R7m2A+9FTGBAKiyTEF54tcE4TVdUpN0tLKnZG
5U3i+h3MOgkTsLZB2YNiHE/B56VwHbF8jkLPXdXK8pQiwzodNUUBisYG25P2cgXt
1YuTROdsmIlwND3PwNOXdae5qpe1jiDqlf2a0LlnTdtTviTaazYiq00o+oc4EDFi
2kiy6IiMw3VA6+F2pSD5yPYkt0FvKIMotlKrVEKYiVpf0B9yd92zMxGW1etlShez
Ququk2dsL5ZaNDX+t+hV0CZQvRbBZxjVPMuwK7Xpqhx/duZb2sogt0xI3ElUsVVc
pQJ/GNV46sTYZXSdx5nlNxE0dISALFMCnHQXLZS76oyMnocSenhH9CtwuMi3Qtv7
qxRzd5gvAb6j/qX2l1lJhMhW1TU+rAQZnTQzM1KtjXUQDr5xb/wBeFR1zOeRnnmu
uqlp7YAgAewXrn2omv4+jM6RdvjNuzxECZ+/3vDGRA3f1pj7mVKC6hPfYz/05G6A
ODhZ4TaKs8kpx2ZggJmqtfzeBdZQOV1q56iwe/DE7K2Rnm1C28xvxR6G+EbU9DZU
DUzBt4c73+8cvN759vVezd3KslLRzFBoScqJNqX3TDduBOGycz0zTyYTo71llifF
aswSxUfL9dKPvEhzzl0SXFM/HvANyiMermdzkfaKOIXT8v+k2eeRlkl99CnZWHzY
R5lDMVtwTth5OjjSbhPa2q6DQlY5uSgkRL9KHzTB1RDLc0h3xn3a7pSEbvDwNTnB
ZmWEDRSbsV3PouAI8QZwMdlKASO2YCl4nwZVku65lHQOuQf4zUlnLVdUE1IOhgrQ
n3/aUwY8bsZ7j1NrzUulUvQbCRYCquXEtTXasvlBpLR0mNNnEDBvQ6NFmVY/L8gt
/+AazJKI5DFggfTC3WKSP7qFLfltsQfQ6fAXTiWAQG6y4kYEQ6Ko0JbZgbtGhF1H
oGi1zHNkW5moVpSfLX8fcQ6iQG4LGmEpM05JOrPrquUVEd7RjkAYvSSkMH4ZDIf7
gC4GiS1U8XTLENaYERTLra5UZXJ+HTGn1F759CsRSGQ1wg99TxJ7+EddSxsiDwV5
bqDtjQB0KZ2n+Y5DugMq/RNsaCxtqdKG+PprITacl23nZWvNebnqvGx3DKI3+dTU
8INPRk5dhqKylj7xlrVJtsonXJpZC8yxrK2mX6WM4avlPrvnKtwdj4d42CMGKooQ
I8ApEHXcB/KnMUYFwbqLQrlJp4govqYEx1IdMh3Lk6E+7tXX8+lRJgWKny3rgebY
DrTlNAbCs1VCqYBT840aQjuPdh6pln4UZcMaMJjNJ6wHzE2Y3PYsgVMuWTRXET/N
WUYNWB9vMLnhkLYg642n535/lYrLIDVpEtp0qAh9KFXJwmJCXT6jjmUbRq6GAvcr
wZ/AwHxyK4IhbqnBFMsDrWpsbOKjyKT0L82b9pjRF25Iy/xvDhFBfMohIqcVXEdm
Q+sMCI9xw2M+WxYbStxy88JhD3W0jxPuXkIlmRCF9XWWnBJeVFiIGrI+xF0821Jh
W0XoNB+ZW8mhQqVsAkqeUbkIuqVDy4IdIQt46ZWu90F04w+jceiHYzyDkFoZl5xF
ZSW/f/D40FdJEjhX3daZ8RfuSUNOAedKMvvc3ie4Cfs+vyUT0Hq+3/aaqPi9CY3a
TWdMYZWOjxUHYTeWQdAknAwDAFX7Nfl1/E0NN1ZqwMkmn0zFNa3Iy6/JggrTfFha
okzv8YIJz9QndDKLq45VVRWaqcjA6vLytkp8sn0WCRH3Gdw+mwtWKLOqwEFTGwda
RXTGSG0M/UrbuhU6fCcT0n1NmJMQV0Z93qaLTXDtttrTV8pkkrNlpScoFYsFXUXi
Pcg2Bca2ePTxnKJFif+Sx6AN23u5x8q/o0mF9mSdR+PJNHONoDzXPbxkn34UX7gn
8WXy8iOP4ouSxOncjEhHUE4MdIN+YNTT+2mLJQf37jl87O4OPrgxqJpgiErHkZSJ
kDvUNzuDOpegbeWiUmQrwsjWvMFq3qCld3fEvIXgvOOOzRvk5jHKxcu+YHlo69px
5Y0Tzx09gTbahbA90HIaGAZg5CTr5wxCw1P6Xgee/ZGKDdqHYCSM3PqfAUOncGr/
QBmDOaWMbqh6VkaDGROvHDUmlaWtTz+D+PPpYJTvq0wHqW8Xz0OQ6eEiny9E+Jgl
xIo2vAl700w5enrbSh1exjPOaTjRBaoH9wBBVaXsalNyp9+XwTGd468ku0yutWJh
JcZFjoPI+pGnM2QyesxHTlf6UUI57fJNgwcit4oVunpCOOrNg7LS8E29071jqPNl
NMBLWnZen8GzCsWrVlb09j5I3BiBoaAe5KDMWVAWVekSqk+pr9AtwuwsMveIlGif
Y2BYJxPfEy3mbJiawsaYqPng9i/pO1j/ql7jEXFk4OTcZO8BD6HZyGju3VjrgCWn
RfziXD86Ei1Dq2CAf0nJzcC0X8mDs2hlmkCl6d5KMf9G1DDCg73WxJao8ZKt2ciZ
dCVMxS767ZLuM8LT6Aznr25BbcKnEYlvkJlyrIVij85jUwVZqjiG02L0Ymhtl59k
dVO4R5NC7raTO12WPJ3LHx5N7ksctpLKoeGbo13M8PX33x6+rGJGn65nIvLOKQQl
wmTSH7owt/XiXTXucTLTyD+agEAGdUCcoqTfSSizPPGmJkr8wb9i+1aHbERiDL5w
D8g569B8R6ZlrqxQrVVSjU5nMmiUUWpl+AdACV4PnboJVVHiu5kg+US05tXJwdbj
n85eHR36O8cH/vd7J6cHR4fEna7bT/2gC3YYXoPzFU4oTmXlULvVLREgZPTSTvMg
YSRXY76fvHZ1b2Nxa5sVLYUWaAJ5hp/Mp1oZVjIDLAepyAGm0yK/PNRvLqNcd2of
BnyYC++q//Zdaf+NH+v+P/xGxd67/If7eOD+v1a7s67v/2vi+9bq+tr65/v/PsXn
YNwPb7ZAK2XTCYjt6vO//6mevDyVWWrPeldpEsfZswldLqi++lGaPUuT3jPVrXdV
hWWeROEVRhAS+KLM3dZKq92ugj4eiOWeWE7o2eC6uLioHypruCki2sBYotXcane3
4I9l5K8KQ1leXjaVN03ldntrdX2r2eXKCNP+UB/tzTVvtdkSVIBwuADgw0dn3KJB
n6UrAayjcb8u9pSiqi/4l3GWwv8LnvhFP6z0Ft41wMR4RHM8+pCEA24uHx7f/Ab3
wjBqkRIA88i2hAK09CAgLcYJjnVhqMLEQuZLESRJcCtDi+kj0KT6BJn+cpGTXHrA
90dKNkqm45XLT8ywJRjM4N32SrdrsS4+ChtvZKwSaJX2mjgCjciM3Nla3dxqbkpG
JpDIf2XtnsbSm5teq9kxHE3PXcXQx7do/6BZaato4bNngPYVG5jbbl0qw1oqU+PB
imQLWbWW7gH3Ko7f5zuvUvjx+yCJ4iluwVLcHrPD4F+wU/mUzuyu98FoVsAkix3b
846U7X1iDisiMJPBWmubDofBs7ARx4kugqu01sVhfCU5bH2r297qdDSHIUxkiJJ2
RQ5bu4fDWmstr7W2ZliMCzYUj6HrMLnEo1FtzELDsyTgF8kJqxgm2o9uphMjJ8Cv
llXRtRbOI7PEktXa4hniLgSNjlIaXaRQQhnAvAN0CY4Lha/TbDrAw820xLCArpMd
Z0m9wbgpGKMgGksYbPeCK49lvo/1XHZiSdkffnJ5Veh/NjNtuLy0ISykLU4ypSiq
9sNzyUitrU5ra9VipA2bj6xWT2UjD1Q+PQLdj//znb97cLL38uzo5Cc8clJ56X/7
9uD12cEhPR3/5O+fHP28B09z9PTyaHfvZO/06O3Jyz2aqaDH29s0lXdShsgDZUhS
OmUHri1zrNde1fz6t/r28p1DlYM3x/6ro6P/3I9IGSP9W0JJ9T6TidqttsNF8Cw0
ygUmApliW20ghza2uquagxBYgYVK5dD6fQzUXfVa3Q1LDkEBNFbzeseUpyvJlmZr
G3j3B4oWow+vPLHIEQXam7APd6tUxyWhwrqcirZE0okjxiRrbtMVbgRWIfTHV1YX
bvTz0HClNOrpxIoFARcxwLIwHhSXb6KDjM2CIYk6Awp7/3VcaygUVXgT25+amwBH
YRZQuBIk7xVVeA0cQgdJmg1qbCeVXnHJDFi5WwUVQHMyZRY8NwHNjNOCbdN0NqoI
6MqOhyOsWbQsJ+aseYwTUeNmdOte7qrEQYDrW9G7kDWKRTLoIS8gyS6xl+uQcg77
Mc985Q7/URx4T4asjoCZYetWc+7oXSaelRurmd6J7OjB6AxZWY9f3Jm+TR+57jE8
miOlchWsVg05fs1ZNLM7XO/Kud+iBKfcuzubpebMXpBDib/DIjPX2yPY4E6uAGcB
3PEaJYElhG0oKWMWiv8gjWTWjgmbAoxSH7cDyq5r+7gbG157s6PVX02u+1qQXFzh
9yRt8Vcbv8KbKMMEwxrpQnjs0eWK/KrnX+GWh3rIkqAXnge997IybsHo2vSgq/Mr
qz4QJL+8i6LEFlTYA86e1rJlo99obngbrTVlZ2jPYl9eeU8Ku/z+erlhzoFuT3BG
stodp21wMktl44KCX7Qv31ONTBdkS6iIqXqNO7J+pvpaLIHvD5L493AsofND3VxU
aflOr/GqGjzhzZcKogKS+xxgJatdTHkj6g8HZ68ERYHf7B2e7e0Sn2x0W95Gd1Px
yT9LOpsUH5uMdv1/kaal7NkB9ux2tBnMu4J4i8Z4OsF+K2jFVGQ6qtrGhCEP+pT6
btHGXDSp7nWmbG15ywvmJWKawYRGw0ktfecEr9ACEIWV3vx4g2HjMK0XtjcaqlFe
vBHvbALvbK5rGfPxx6V3Wz7y6EpnsbvubaytmlmsjLSlOGPwMNSRuo5LDpR3V3Ek
NEAYwPKLjHdh1JisLW6Ea2fdE/9zF1sc+Gk2vU30wNQU/KNIFcj9WNRK6bfR8TY2
Nq2gVXvdazVb2pS3dmilXb8kzLkg0rnW78GQFi1mQi1RnoyjUqwgWN4wN29spWSX
T0pcBwVAewi/eWLMlqO2ZygZ0QaqjXPHQrvv8mwxMRbfLJPL7OvlbCYlsWxUpT2S
M1OkT4OnC10Efpm8w3QYzKWh20PQOMErJaZJKi9/0c6LnUdRHIZn7oOb01a/kwvA
WFD65G980PU3POhKhIO/+ZQrj1dNAvGDJrcioTuNv0liIJJcX3EvGXw5FHL3mped
ZUAgeIqhpifHnZ2cqakSyZ4gjdjrKLFR9bsZdmrBMt0ypqkWcMd4O1ddl97J7wIh
1JsS18a6hg8f74pEmLNSsKiFTEF1Tfkcv5Y533cF7n8+i/tNonhhHuUdsxKeTCJ5
tfP9nn96tnNW1oV1eyFCM7mopcmKihGknYEiif6hI26KONx8fh6vAQkyeRHHvKzG
ywLdlrlT/+B09+CkLt+sAGuPKJdcz7/8kSqVA84/7YE2ChQsQski3eij7mWQV7zQ
+TbwzWCJeAwGpHXKqW4yAKs4aBTCeMdROhLX9AMEeBEy3peJF5aHwZAPLEXZigZz
Eo7iKz7cTaKC0qMv42GfLgDEoIjDnPjbRcIHT9gneBqMhCt/g6SHt6WbyzqHAV6G
qxxHW+jshsMZstOSNSUcfqcZghKpWQbyTy68H8fX4hrR5EvaoTf23eEdXcFDST1A
4+UxZVzC6oDxA+ooQpXAJFRlNlKJDJcsZv3cwj3S03jO6kjbvUspP1Yaaok+uLtf
h1KiWGKZyI5a/EcVqccehdKoZvUWdXm+Y7e7Mo1pZYiWLfKZ2pIxyqsCzherG8xz
t3vW0IGVDoXj/buTpC5416O+0+7HaYjJvbSeaKHXQV4F02GmI4sN4r9AeiIruAIJ
GLTgjVq6TiGdxGP9ExLsZbGE8kha1K+iQLk1+ofHGgSj1HjbXPM2Wy3Lw7Z+dq0X
pKEfv687nh/9Iw/Cbrv1iWJk7jseH2/GWI6n3U/RZawKUfQaQdMMy3jV+W0H9hzR
1sXbe9EgRqpJP3LigzmrQjDyRu8xX/ANBbIp3fqL/dAN/Vhe9GjJx9GWuEy/h/cj
SkaGInX62aoG2mmM6unk4PTl0akF2NUqQkkWCog3W2hHr3YsH/4TTM7SzJlhTscV
yc/4M5C4MuB/tWdiNkw+/eQ6gYKJFHP/0ow70lBHvby8FM2J27/CKWWLerO14W22
V629lWanhYkFZpdXjj4FDdf37eHLX4egcj65qXnYUXqKwMZxRHmr6W61U06mvDdK
TXbxojQhZp0qz12VVjrmNgiy1ZY95i6OeUOnA3HQQqbBM7tpXNQOtYx06xlzLBzj
GBUGL1W3aVfye00m+Gl5GFYLo87mpPPDZw51FWMrzjh7bx+Ale5BzR2OupU/EHhn
K/08m65pDTXVDka5fVUZ60Eap9jgyTUL1xvR0lNXG8lx2IFHqZ8fdgatBRUpRO/V
6tK9y2v0I6nSNbUqzkx8o625LWGNiw8tyO4cv9OhFv9AH6UL2+atYZycN8JHVqTl
IK7BwNY/tsGv7bXlkEt1O58T0HmXT1ksUjLm7GW1TIkEzi8COD8YoRlR8MbQ2aU8
mI5xJWSodHouQ6l4Te112MetlqiPPFfj6GyN2tKBVhmJxV8/BQuFfjsNY7kMgPbr
ZB3+VbTShb8Owm7D3khutTe9VqvbtYJnvA5mLAHjVdEVPpyvkmN9IYmkzVjfxIVy
69zZBsnbrarZrMXuRJaMR/DI5V4r7HjOXvCl673UxfgrP0W3ZJjJbuoM+/5fo3v6
wHOdmaHj0Z/7RoizWybR5KLHt6whhf4BD601HhBzwolt9eLJrRPGtdIWimLOlnBs
GaBhAOv47enB4Xf+28MDTGWxguNvx1HPnFW/MlFw2a+psTemE36mYOdUwYN25Rq2
1Wx2Oa5rLbX1Niw1azuyYu9uYrdyMZnoEorFK4GPAeaV4d6BTC7TK6ySiw5LEVim
eVFCOQxvyqmNOjdf/EXEh5SQlrWWz8xWw/2u6tV/pZrga+zUYCROeF90Ic4kEblz
mHYU9MAAj9NLqTboC3jqdBKG/elkS4QBuMZaRuMvF9DF2ZQ5iseUwbUtZ8pOp+O1
Ot2m8mDx5qsHd/PECf74wt7hyz3x8ujt4VneKbViJnOiuCfKbKCMfsdLQTeDtn3M
Zr910GtKt765pdsyp6294bXWjLP3f2QctlOV86geMbLSGeu2MfOrbWvs9SZ4uutt
Y6pbEeCRlCAy2Mx2Ojq/2uncshJAcgvuLxjMtuDB2GU8zSTnP2QUj2ZYn9wasyac
33vJRZSsaLrSSTJAtGV5Ru4laDb+QqiN9NquCXdewv9ZrNamvGakTrYaHXv+qt+o
zRD3aFi1Njes7e3CpJBHJU+AzznqTLvZM39VXZjQuutr86uCS03JL6hv4J+Nexjl
fpysiOQ/iVw5ATe7mNHYMgSUwcBKydZ5yd65/jUya+PcxtPKCLC1rrFaZm2lP3X7
StJ+YxNov2mZ1R9hOPCP1jsff1xl0wYz1oFpazWthICy/ShhSQK9GcVUmZPCwJa4
5gT2rBwBuRPPO/C5LXhzT4BrEfDEbG547Xaz6xphHxNhT+hJmnMnyZm8vzCg8hnZ
aMKMbLoz4mxgPiUl5QnLqpT78sxXdBb/YnpIWe6FPn8vBWB7HWRKe8Od649LCjtZ
5OMTpJia8gBZSjkGSARkWjW6S904ZJPEDnjYG0fYnfQGNfpF2tmEM/cZzWCkh0lU
zMRRy7sD9hL8o1P5/r2x3Jel9cgRlc5WdxMk7lrTWt+5n6hwZJcVkJlzNtyKrGXG
otfa3SwDmWi9jjpu3dZxHwUTi5Az8Sml1Hp7DfBbdc68tlse/GOy9fh2DTwbVcdc
cH0aBqxN/Tef4heURYZZJdMk3C5pq8/O1PAGBvn349vaJ20Ygl1SCmepAEdZ+dYu
0wwMyGuSz2Qz69ODr6PzZyk4lZ/+JLXV88zD1J015yx1Z00oXHHOLQiVTXNCsNnd
am7oM1oVgoL84FZ/yokc0CwqjCoE/u5P6kdjH3xytaPxB/KqTJHABBTlrm9V1eHi
/WhIP6I45ZSzlJNCOPsivAGaoLTPVGIFp7Bwogv/TDAsIwaURvijflG2QL9CNQl7
Ed0GmYYr4gcMLKexGIdhH10b+XaZXEIE+CboeRoQxkf0D19RoFlFpWMzAGyWhCuy
EfAfvRgGGQx2hNJrYRT0Frbke1kHiYAb7nTXtLx5KUqhtA7/N+zKvHXP4S9ZjP7J
I+ARzdL7AfIjVPHwH6IC/7gxQqCmVd0HwlaVgPyFSVYziZ/X6ogHgUDWQgbRGan/
dxlkX/6m88zpXtFQ8DKfHyJ3my9lqP3pOR1exmuervgmcRtzClGlGlB+GCX5WVvq
B9souNWLp8M+R6ffnrzWcJB44U2Av8ZnseMTWOMTs8W/fU3G58/nz+fP58/nz+fP
58/nz3/N538B2FEpBwCgAAA=
--11364638-0-3248466697=:9007--


From skip@pobox.com  Mon Dec  9 23:18:42 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 9 Dec 2002 17:18:42 -0600
Subject: [Python-Dev] os.path.commonprefix
In-Reply-To: <20021209224815.8F7D94B52@bespin.org>
References: <20021207160218.1528F49BA@bespin.org>
 <15858.8941.920830.141997@montanaro.dyndns.org>
 <20021209224815.8F7D94B52@bespin.org>
Message-ID: <15861.9426.521685.951820@montanaro.dyndns.org>

>>>>> "Armin" == Armin Rigo <arigo@tunes.org> writes:

    Armin> Can't we deprecate the thing and move it elsewhere? (where?)

How about just adding commonpathprefix?  As I mentioned before, there's an
implementation in Tools/scripts/trace.py.

    Armin> In all cases, wouldn't it be worth a documentation note?

The docs for the function state, "Note that this may return invalid paths
because it works a character at a time."  I suspect that's sufficient.

Skip


From python@ragnark.vestdata.no  Tue Dec 10 03:45:39 2002
From: python@ragnark.vestdata.no (=?iso-8859-1?Q?Ragnar_Kj=F8rstad?=)
Date: Tue, 10 Dec 2002 04:45:39 +0100
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
In-Reply-To: <3DF46D4A.2070902@lemburg.com>; from mal@lemburg.com on Mon, Dec 09, 2002 at 11:15:38AM +0100
References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com>
Message-ID: <20021210044539.H26400@vestdata.no>

On Mon, Dec 09, 2002 at 11:15:38AM +0100, M.-A. Lemburg wrote:
> > The dumps-method in xmlrpclib has the following comment:
> >     All 8-bit strings in the data structure are assumed to use the
> >     packet encoding.  Unicode strings are automatically converted,
> >     where necessary.
> >=20
> > This doesn't work very well. In our particular case we're using latin=
_1
> > as our default encoding, and we're using UTF-8 for the packet encodin=
g.
> > We can't really change the default encoding, because the sql-modules
> > transfer latin_1 encoded data and we can't change the packet encoding=
 to
> > latin_1 because the xmlrpc-client (php) doesn't work with that.
> >
> > The attached patch changes xmlrpclib to convert strings to unicode us=
ing
> > the default encoding, and then convert them back to strings with the
> > packet encoding. If unicode is not available it falls back to the old
> > behaviour.
>=20
> I believe this is overkill. If you need this behaviour, subclass
> the Marshaller in xmlrpclib and add your feature to that subclass.
> Then replace the Marshaller class in xmlrpclib with your subclass.

Well, we replaced the xmlrpclib.Marshaller.dump_string method from our
application. That works as a workaround for us, but my point was not not
to merely make our application work but to fix this problem for other
python users as well.

The library makes an assumption that is (IMHO) just not valid. There is
simply no reason to assume strings use the packet encoding.=20

Why would you not like to fix this? Because of the performance? It would
be possible to have both functions available in the class, and only use
the encoding-convertion when the encodings are actually different. This
could be done with no other performance penalty than a simple check when
the encoding is set. (The constructor?)

I simply didn't include this code in my patch because it would make the
code harder to read and I think most people use ascii or latin_1 for thei=
r=20
string-encoding and UTF-8 as their packet-encoding.


> Aside: xmlrpclib should support subclassing the Marshaller and
> Unmarshaller more transparently. Currently, the two are hard-coded
> into the rest of xmlrpclib without the possibility to provide your
> own subclasses without tweaking xmlrpclib from the outside.

In principle I agree, but it should not be neccessary to subclass the
Marshaller for most applications, and tweaking it from the outside can
be done pretty easily in python :)


> Please post patches using the SourceForge patch manager.

Didn't you just write that the patch was overkill and you didn't want
it? Do you want me to post it anyway? Or did you just mean for any
potentail future patches?



--=20
Ragnar Kj=F8rstad
Zet.no


From martin@v.loewis.de  Tue Dec 10 06:10:37 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 10 Dec 2002 07:10:37 +0100
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
In-Reply-To: <20021210044539.H26400@vestdata.no>
References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com>
 <20021210044539.H26400@vestdata.no>
Message-ID: <m3bs3uieua.fsf@mira.informatik.hu-berlin.de>

Ragnar Kj=C3=B8rstad <python@ragnark.vestdata.no> writes:

> The library makes an assumption that is (IMHO) just not valid. There is
> simply no reason to assume strings use the packet encoding.=20

There isn't a reason to assume that strings use any other encoding,
either. Reliance on the default encoding should be avoided as much as
possible. Users wanting to transmit non-ASCII strings with xmlrpclib
should pass Unicode objects in the first place.

Use byte strings to represent bytes, use Unicode strings to represent
characters.

Regards,
Martin



From python@ragnark.vestdata.no  Tue Dec 10 08:25:14 2002
From: python@ragnark.vestdata.no (=?iso-8859-1?Q?Ragnar_Kj=F8rstad?=)
Date: Tue, 10 Dec 2002 09:25:14 +0100
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
In-Reply-To: <m3bs3uieua.fsf@mira.informatik.hu-berlin.de>; from martin@v.loewis.de on Tue, Dec 10, 2002 at 07:10:37AM +0100
References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> <m3bs3uieua.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021210092514.L26400@vestdata.no>

On Tue, Dec 10, 2002 at 07:10:37AM +0100, Martin v. L=F6wis wrote:
> Ragnar Kj=C3=B8rstad <python@ragnark.vestdata.no> writes:
>=20
> > The library makes an assumption that is (IMHO) just not valid. There =
is
> > simply no reason to assume strings use the packet encoding.=20
>=20
> There isn't a reason to assume that strings use any other encoding,
> either.=20

Isn't that what sys.{get,set}defaultencoding is for?

> Reliance on the default encoding should be avoided as much as
> possible. Users wanting to transmit non-ASCII strings with xmlrpclib
> should pass Unicode objects in the first place.
>=20
> Use byte strings to represent bytes, use Unicode strings to represent
> characters.

So perhaps it would be better to change the postgresql-modules to return
unicode-objects instead of strings?



--=20
Ragnar Kj=F8rstad
Zet.no


From fredrik@pythonware.com  Tue Dec 10 08:40:26 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Tue, 10 Dec 2002 09:40:26 +0100
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> <m3bs3uieua.fsf@mira.informatik.hu-berlin.de> <20021210092514.L26400@vestdata.no>
Message-ID: <005f01c2a027$ca50d920$0900a8c0@spiff>

Ragnar Kj=F8rstad wrote:

> > There isn't a reason to assume that strings use any other encoding,
> > either.=20
>=20
> Isn't that what sys.{get,set}defaultencoding is for?

Tthe default encoding stuff was added for experimental purposes only; in =
the
early days, there was some disagreement on whether the default encoding
should be UTF-8, ISO-Latin-1 (8-bit subset of unicode), or ASCII (7-bit =
sub-
set of unicode).  We finally settled on ASCII, for a couple of reasons.

Robust application code should assume that the default encoding will =
*always*
be ASCII, and use Unicode strings and ASCII-only strings for text.

> So perhaps it would be better to change the postgresql-modules to =
return
> unicode-objects instead of strings?

That's the correct solution, yes.

Checking for chars>=3D128 and returning Unicode only if the string =
contains
non-ASCII characters is also a valid approach.

</F>



From brian@sweetapp.com  Tue Dec 10 08:48:26 2002
From: brian@sweetapp.com (Brian Quinlan)
Date: Tue, 10 Dec 2002 00:48:26 -0800
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
In-Reply-To: <20021210092514.L26400@vestdata.no>
Message-ID: <002101c2a028$e803e2e0$21795418@dell1700>

> > There isn't a reason to assume that strings use any other encoding,
> > either.
> 
> Isn't that what sys.{get,set}defaultencoding is for?

If that is all you want to support, then:

def dump_string(self, value, escape=escape):
    if unicode:
        self.dump_unicode(unicode(value), escape)
    else:
        self.write("<value><string>%s</string></value>\n" 
		% escape(value))

dispatch[StringType] = dump_string

If you have changed the default encoding (not recommended) then
everything should work fine.

> So perhaps it would be better to change the postgresql-modules to
return
> unicode-objects instead of strings?

Maybe. 

Cheers,
Brian



From =?utf-8?Q?Martin_v._L=C3=B6wis?=  Tue Dec 10 08:56:17 2002
From: =?utf-8?Q?Martin_v._L=C3=B6wis?= (=?utf-8?Q?Martin_v._L=C3=B6wis?=)
Date: Tue, 10 Dec 2002 09:56:17 +0100
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no> <m3bs3uieua.fsf@mira.informatik.hu-berlin.de> <20021210092514.L26400@vestdata.no>
Message-ID: <000801c2a02a$00edf2e0$ad25e8d9@mira>

> > There isn't a reason to assume that strings use any other encoding,
> > either.
>
> Isn't that what sys.{get,set}defaultencoding is for?

No. If it was for me, those functions would not be there in the first
place, and the Python system encoding would be ASCII. I consider it an
advantage that setdefaultencoding isn't really there.

Byte strings are used in many places in Python, e.g. when writing to
files. You can't possibly know the encoding of all files that you may
encounter in advance, so the notion of a single default encoding is too
limited in practice - as you have just discovered.

The purpose of the default encoding is to define automatic coercion
between byte strings and Unicode strings, e.g. when comparing or
concatenating them. This coercion is necessary primarily because of
string literals; if they were Unicode strings instead of byte strings,
coercion and the default encoding could be deprecated.

> So perhaps it would be better to change the postgresql-modules to
return
> unicode-objects instead of strings?

I'm without context here to answer the question. If you are talking
about a character string type, and if postgresql has a well-defined
notion of what the encoding of character strings is, then yes (except
for backwards compatibility concerns). If you are talking about BLOBs,
then no: these are byte strings, and don't have any inherent encoding.

I don't know how SQL deals with non-ASCII strings. If (as I assume)
there are no clear specifications, it might be best to introduce the
notion of a "table encoding" or "driver encoding" in the DB ABI. If this
is not set, the driver would return byte strings. If it is set, the
driver would return Unicode strings, decoding the data it receives from
the underlying database appropriately.

I don't use relational database or SQL regularly, so take this advise
with a grain of salt.

My 0.02 €,
Martin



From Jack.Jansen@cwi.nl  Tue Dec 10 10:15:15 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Tue, 10 Dec 2002 11:15:15 +0100
Subject: [Python-Dev] Complexity of import.c
In-Reply-To: <3DF4D1BB.9030400@interet.com>
Message-ID: <46B467C6-0C28-11D7-AE16-0030655234CE@cwi.nl>

On Monday, Dec 9, 2002, at 18:24 Europe/Amsterdam, James C. Ahlstrom 
wrote:
> I tried my best not to break the Macintosh port, which has
> a lot of special code.  A replacement import.c should do
> the same.

Actually, if the new improved import has the right set of hooks then 
the macintosh stuff can go away. 90% of it does basically the same as 
what zip-import does: it allows you to put files on sys.path and looks 
in those files for modules. The mumbo-jumbo about interning sys.path 
strings is so the code can cache whether a sys.path entry is a file or 
a directory, without that normal imports were slowed down by 30% or so.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From mal@lemburg.com  Tue Dec 10 11:04:53 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 10 Dec 2002 12:04:53 +0100
Subject: [Python-Dev] Re: [Patches] Patch for xmlrpc encoding
References: <20021206025220.E17658@vestdata.no> <3DF46D4A.2070902@lemburg.com> <20021210044539.H26400@vestdata.no>
Message-ID: <3DF5CA55.1070505@lemburg.com>

Ragnar Kj=F8rstad wrote:
> On Mon, Dec 09, 2002 at 11:15:38AM +0100, M.-A. Lemburg wrote:
>=20
>>>The dumps-method in xmlrpclib has the following comment:
>>>    All 8-bit strings in the data structure are assumed to use the
>>>    packet encoding.  Unicode strings are automatically converted,
>>>    where necessary.
>>>
>>>This doesn't work very well. In our particular case we're using latin_=
1
>>>as our default encoding, and we're using UTF-8 for the packet encoding.
>>>We can't really change the default encoding, because the sql-modules
>>>transfer latin_1 encoded data and we can't change the packet encoding =
to
>>>latin_1 because the xmlrpc-client (php) doesn't work with that.
>>>
>>>The attached patch changes xmlrpclib to convert strings to unicode usi=
ng
>>>the default encoding, and then convert them back to strings with the
>>>packet encoding. If unicode is not available it falls back to the old
>>>behaviour.
>>
>>I believe this is overkill. If you need this behaviour, subclass
>>the Marshaller in xmlrpclib and add your feature to that subclass.
>>Then replace the Marshaller class in xmlrpclib with your subclass.
>=20
> Well, we replaced the xmlrpclib.Marshaller.dump_string method from our
> application. That works as a workaround for us, but my point was not no=
t
> to merely make our application work but to fix this problem for other
> python users as well.

The first point I want to make is that non-ASCII text doesn't belong in
strings, it belongs in Unicode objects. I don't think there's anything
to argue here.

My second point is that I would like xmlrpclib to be more flexible
w/r to using user-supplied marshallers and unmarshallers. The
reasoning here is that applications will want to send custom
types/classes over the wire and need to provide custom mappings
for these via those adapted marshallers, e.g. a product I'm working
on uses mxDateTime objects to store date/time values and passes
in buffer() objects to signal: this is binary data.

> The library makes an assumption that is (IMHO) just not valid. There is
> simply no reason to assume strings use the packet encoding.=20

It is valid with respect to what the whole standard lib assumes:
strings contains ASCII text data or binary data.

> Why would you not like to fix this? Because of the performance? It woul=
d
> be possible to have both functions available in the class, and only use
> the encoding-convertion when the encodings are actually different. This
> could be done with no other performance penalty than a simple check whe=
n
> the encoding is set. (The constructor?)
>=20
> I simply didn't include this code in my patch because it would make the
> code harder to read and I think most people use ascii or latin_1 for th=
eir=20
> string-encoding and UTF-8 as their packet-encoding.

A correct fix would be to check whether the strings is indeed
using the xmlrpc encoding. That is IMHO overkill, since this is
something to think about when designing the application and
not a runtime check that's always needed.

>>Aside: xmlrpclib should support subclassing the Marshaller and
>>Unmarshaller more transparently. Currently, the two are hard-coded
>>into the rest of xmlrpclib without the possibility to provide your
>>own subclasses without tweaking xmlrpclib from the outside.
>=20
>=20
> In principle I agree, but it should not be neccessary to subclass the
> Marshaller for most applications, and tweaking it from the outside can
> be done pretty easily in python :)

Sure, but it's not a clean solution :-( I've done that too in the past,
but reverted to writing my own code for the dumps() and loads()
APIs to be able to use subclassing to implement the above mentioned
mappings.

>>Please post patches using the SourceForge patch manager.
>=20
>=20
> Didn't you just write that the patch was overkill and you didn't want
> it? Do you want me to post it anyway? Or did you just mean for any
> potentail future patches?

No, I meant that patches should always be posted there. SF is what
we use to track patches and bugs.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From erik@pythonware.com  Tue Dec 10 11:54:34 2002
From: erik@pythonware.com (Erik Heneryd)
Date: Tue, 10 Dec 2002 12:54:34 +0100 (CET)
Subject: [Python-Dev] anyone tried to use optparse with something else than sys.argv?
Message-ID: <Pine.LNX.4.44.0212101248380.29743-100000@brain.secret.pythonware.com>

hi

snuck a peek at the newly included optparse module yesterday and it seems
option parsing will be a breeze in the future.  good work!

however, imho, it seems a bit too focused on initial sys.argv parsing.
for anything else, you basically have to subclass OptionParser etc to do
anything useful.

for example, you can supply your own argument list to the parse_args()
method, but the autohelp/version/error reporters still use sys.argv[0] as
command name (probably not what you want).  the way optparse gets the
command name (a global function) indicates that you're not supposed to
override it?

also, even if you tell OptionParser to use your user defined Option class
(as explained in the docs: op = OptionParser(option_class=MyOption)), it
still adds standard Option instances to handle autohelp/version.

per default, the autohelp/version/error handlers call sys.exit().  the
docs says that if you want to change this behaviour, simply subclass
OptionParser and/or Option and override/extend the corresponding methods.
ofcourse, this has no effect, since standard instances are used.

also, supressing the actual printing of autohelp/version/error requires
subclassing OptionParser and Option (or sys.stdout = StringIO fiddling).

so?  some minor glitches and (imho) annoyances, whats the deal?

well, to me, it's pretty obvious that the design effort on optparse has
pretty much been focused strictly on initial sys.argv parsing and while
it's extendable, other use cases (embedded consoles etc) seem to have been
given little or no real thought.

the things i mention is probably easy to fix:

* make parse_args() take the whole "sys.argv" list and use the first
element as the command name.  breaks backward compatibility, i know.
why was sys.argv[1:] used in the first place?  the getopt legacy?

* add parse_args_ex() that raises an exception containing the help/error
message instead of SystemExit (SystemExit containing something means
something went wrong, don't it?)

* (simple bug) dynamically populate the OptionParser using the
option_class and not some pre-made Option.

* (simple bug) replace all %prog in help texts etc, not just the first
occurance.


        erik




From jim@interet.com  Tue Dec 10 15:09:46 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Tue, 10 Dec 2002 10:09:46 -0500
Subject: [Python-Dev] Complexity of import.c
References: <3DF4D1BB.9030400@interet.com> <018701c29faa$75f7b560$f92fe8d9@mira> <3DF50BB9.2080603@interet.com> <029f01c29fcc$29aa3300$f92fe8d9@mira>
Message-ID: <3DF603BA.5090504@interet.com>

Martin v. Löwis wrote:
> So what would have been the problem with linear search that makes this
> extra data structure strictly necessary to have?

Speed.  The linear search would be repeated with each import.

> I think you can't really cover all cases. For example, your patch fails
> to find zlib in the Unix build tree, as the build directory is added to
> sys.path only in site.py, and your patch won't attempt to import zlib
> outside InitZip.

Hmm, you are right.  But easily fixed.

JimA



From guido@python.org  Tue Dec 10 15:31:06 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 10 Dec 2002 10:31:06 -0500
Subject: [Python-Dev] anyone tried to use optparse with something else than sys.argv?
In-Reply-To: Your message of "Tue, 10 Dec 2002 12:54:34 +0100."
 <Pine.LNX.4.44.0212101248380.29743-100000@brain.secret.pythonware.com>
References: <Pine.LNX.4.44.0212101248380.29743-100000@brain.secret.pythonware.com>
Message-ID: <200212101531.gBAFV6h31663@odiug.zope.com>

> snuck a peek at the newly included optparse module yesterday and it seems
> option parsing will be a breeze in the future.  good work!
> 
> however, imho, it seems a bit too focused on initial sys.argv parsing.
> for anything else, you basically have to subclass OptionParser etc to do
> anything useful.
> 
> for example, you can supply your own argument list to the parse_args()
> method, but the autohelp/version/error reporters still use sys.argv[0] as
> command name (probably not what you want).  the way optparse gets the
> command name (a global function) indicates that you're not supposed to
> override it?
> 
> also, even if you tell OptionParser to use your user defined Option class
> (as explained in the docs: op = OptionParser(option_class=MyOption)), it
> still adds standard Option instances to handle autohelp/version.
> 
> per default, the autohelp/version/error handlers call sys.exit().  the
> docs says that if you want to change this behaviour, simply subclass
> OptionParser and/or Option and override/extend the corresponding methods.
> ofcourse, this has no effect, since standard instances are used.
> 
> also, supressing the actual printing of autohelp/version/error requires
> subclassing OptionParser and Option (or sys.stdout = StringIO fiddling).
> 
> so?  some minor glitches and (imho) annoyances, whats the deal?
> 
> well, to me, it's pretty obvious that the design effort on optparse has
> pretty much been focused strictly on initial sys.argv parsing and while
> it's extendable, other use cases (embedded consoles etc) seem to have been
> given little or no real thought.
> 
> the things i mention is probably easy to fix:
> 
> * make parse_args() take the whole "sys.argv" list and use the first
> element as the command name.  breaks backward compatibility, i know.
> why was sys.argv[1:] used in the first place?  the getopt legacy?
> 
> * add parse_args_ex() that raises an exception containing the help/error
> message instead of SystemExit (SystemExit containing something means
> something went wrong, don't it?)
> 
> * (simple bug) dynamically populate the OptionParser using the
> option_class and not some pre-made Option.
> 
> * (simple bug) replace all %prog in help texts etc, not just the first
> occurance.

You make some good points.

A patch uploaded to SF would be welcome.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gward@python.net  Tue Dec 10 16:32:35 2002
From: gward@python.net (Greg Ward)
Date: Tue, 10 Dec 2002 11:32:35 -0500
Subject: [Python-Dev] Proposed changes to linuxaudiodev
In-Reply-To: <200212091529.gB9FTbG19825@odiug.zope.com>
References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca> <20021207212326.GA13707@panix.com> <200212091529.gB9FTbG19825@odiug.zope.com>
Message-ID: <20021210163235.GA8067@cthulhu.gerg.ca>

[me]
> Sounds fine to me.  Should I worry about adding an official
> DeprecationWarning to linuxaudiodev.c?  Or does that wait until 2.4?

[Aahz]
> Given its lack of official documentation, I'd vote for DeprecationWarning
> now.

[Guido]
> +1.
> 
> Somebody check this in please.  (But make sure that running the unit
> tests doesn't issue the warning.)

Two points:

  * ossaudiodev isn't 100% compatible with linuxaudiodev -- does this
    affect your decision?  (Currently the incompatible is limited
    to subtle behavioural stuff, but I plan to change some peculiar
    method signatures -- setparameters(), in particular, takes an
    'ssize' argument that serves no purpose as near as I can tell.)

  * if I remove test_linuxaudiodev.py, which should be fine since I've
    just created and checked in test_ossaudiodev.py, then the warnings
    from the test suite won't be an issue.  OK if I remove it?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
I am NOT a nut....


From gward@python.net  Tue Dec 10 16:40:48 2002
From: gward@python.net (Greg Ward)
Date: Tue, 10 Dec 2002 11:40:48 -0500
Subject: [Python-Dev] anyone tried to use optparse with something else than sys.argv?
In-Reply-To: <Pine.LNX.4.44.0212101248380.29743-100000@brain.secret.pythonware.com>
References: <Pine.LNX.4.44.0212101248380.29743-100000@brain.secret.pythonware.com>
Message-ID: <20021210164048.GB8067@cthulhu.gerg.ca>

[cc'ing optik-users@lists.sourceforge.net, since this concerns
 Optik's reincarnation as optparse in the Python standard library]

On 10 December 2002, Erik Heneryd said (on python-dev):
> snuck a peek at the newly included optparse module yesterday and it seems
> option parsing will be a breeze in the future.  good work!

Thanks!

> well, to me, it's pretty obvious that the design effort on optparse has
> pretty much been focused strictly on initial sys.argv parsing and while
> it's extendable, other use cases (embedded consoles etc) seem to have been
> given little or no real thought.

Guilty as charged.  My intention was always to support that kind of
stuff, but I've never had occasion to test it myself.  Sounds like I
carefully hid lots of little bugs in there -- congrats for finding them.
;-)

> the things i mention is probably easy to fix:
> 
> * make parse_args() take the whole "sys.argv" list and use the first
> element as the command name.  breaks backward compatibility, i know.

Since optparse is new to the stdlib as of Python 2.3, backwards
compatibility is not a huge concern; anyone switching from Optik to
optparse will have to make changes anyways.  So this is an option.

> why was sys.argv[1:] used in the first place?  the getopt legacy?

Gee, good question.  I can assure you it had nothing to do with getopt.
I think it was because, despite Unix/libc conventions, I don't see the
program name as part of the argument list.  That's a perfectly valid
interpretation, but I failed to allow a clean/obvious way to override
argv[0].

Idea: add a keyword arg 'prog' to OptionParser.__init__() that sets
self.prog, and then make get_prog_name() a method that returns self.prog
or sys.argv[0].  That feels right.  The other possibility is to add a
'prog' argument to parse_args(), but I'm not sure offhand what would
happen to it (and how it would find its way to down to usage/version
strings.)

Anyways, thanks for the bug reports.  I'll use your message as a to-do
list for Optik 1.4.1.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Think honk if you're a telepath.


From guido@python.org  Tue Dec 10 16:40:10 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 10 Dec 2002 11:40:10 -0500
Subject: [Python-Dev] Proposed changes to linuxaudiodev
In-Reply-To: Your message of "Tue, 10 Dec 2002 11:32:35 EST."
 <20021210163235.GA8067@cthulhu.gerg.ca>
References: <20021129182739.GA23142@cthulhu.gerg.ca> <200211300137.gAU1bB209985@pcp02138704pcs.reston01.va.comcast.net> <20021130164818.GA25894@cthulhu.gerg.ca> <20021207212326.GA13707@panix.com> <200212091529.gB9FTbG19825@odiug.zope.com>
 <20021210163235.GA8067@cthulhu.gerg.ca>
Message-ID: <200212101640.gBAGeA032131@odiug.zope.com>

> [me]
> > Sounds fine to me.  Should I worry about adding an official
> > DeprecationWarning to linuxaudiodev.c?  Or does that wait until 2.4?
> 
> [Aahz]
> > Given its lack of official documentation, I'd vote for DeprecationWarning
> > now.
> 
> [Guido]
> > +1.
> > 
> > Somebody check this in please.  (But make sure that running the unit
> > tests doesn't issue the warning.)
> 
> Two points:
> 
>   * ossaudiodev isn't 100% compatible with linuxaudiodev -- does this
>     affect your decision?  (Currently the incompatible is limited
>     to subtle behavioural stuff, but I plan to change some peculiar
>     method signatures -- setparameters(), in particular, takes an
>     'ssize' argument that serves no purpose as near as I can tell.)

I don't care about API compatibility -- as long as oss lets you do the
same stuff (possibly by making different calls) I think it's a good
replacement.  The deprecation warning for linuxaudiodev doesn't mean
it's illegal to use it today -- it means it will go away in the future.

>   * if I remove test_linuxaudiodev.py, which should be fine since I've
>     just created and checked in test_ossaudiodev.py, then the warnings
>     from the test suite won't be an issue.  OK if I remove it?

Typically, we don't remove tests for deprecated code until we remove
the deprecated code itself.  This saves us from accidentally breaking
the deprecated code before it's time to kill it.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gward@python.net  Wed Dec 11 13:49:58 2002
From: gward@python.net (Greg Ward)
Date: Wed, 11 Dec 2002 08:49:58 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]>
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net> <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <20021211134958.GA10384@cthulhu.gerg.ca>

On 11 December 2002, Just van Rossum said:
> This checkin makes textwrap fail for me when running python -S ("don't run
> site.py"):
> 
> Python 2.3a0 (#42, Dec 11 2002, 10:33:42) 
> [GCC 3.1 20020420 (prerelease)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import textwrap
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/Users/just/code/python_cvs/Lib/textwrap.py", line 15, in ?
>     class TextWrapper:
>   File "/Users/just/code/python_cvs/Lib/textwrap.py", line 56, in TextWrapper
>     unicode_whitespace_trans[ord(unicode(c))] = ord(u' ')
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 0: ordinal
> not in range(128)
> >>> 

Weird.  What the heck is 0xa0 doing in string.whitespace?  Oh, waitasec,
0xa0 is non-breaking-space in ISO-8859-1.  But no, string.whitespace is
hardcoded to the US-ASCII whitespace chars, all of which are in
range(128) -- your locale shouldn't matter.

Just, can you run the following commands and tell me what you get:

  python -c "import string; print [hex(ord(c)) for c in string.whitespace]"
  python -S -c "import string; print [hex(ord(c)) for c in string.whitespace]"
  LANG=C python -c "import string; print [hex(ord(c)) for c in string.whitespace]"
  LANG=C python -S -c "import string; print [hex(ord(c)) for c in string.whitespace]"

(On my Linux box where LANG=en_CA normally, I always get the six
US-ASCII whitespace chars.)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
This quote intentionally left blank.


From fredrik@pythonware.com  Wed Dec 11 14:04:58 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 11 Dec 2002 15:04:58 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net> <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]> <20021211134958.GA10384@cthulhu.gerg.ca>
Message-ID: <017b01c2a11e$4b655c60$0900a8c0@spiff>

Greg Ward wrote:

> Weird.  What the heck is 0xa0 doing in string.whitespace?  Oh, =
waitasec,
> 0xa0 is non-breaking-space in ISO-8859-1.  But no, string.whitespace =
is
> hardcoded to the US-ASCII whitespace chars, all of which are in
> range(128) -- your locale shouldn't matter.

$ tail Lib/string.py

try:
    from strop import maketrans, lowercase, uppercase, whitespace
    letters =3D lowercase + uppercase
except ImportError:
    pass

$ tail Modules/strop.c

DL_EXPORT(void)
initstrop(void)
{
    ...

    /* Create 'whitespace' object */
    n =3D 0;
    for (c =3D 0; c < 256; c++) {
        if (isspace(c))
            buf[n++] =3D c;
    }
    s =3D PyString_FromStringAndSize(buf, n);
    if (s) {
        PyDict_SetItemString(d, "whitespace", s);
        Py_DECREF(s);
    }

 ...

</F>



From just@letterror.com  Wed Dec 11 14:17:59 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 11 Dec 2002 15:17:59 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <20021211134958.GA10384@cthulhu.gerg.ca>
Message-ID: <r01050400-1022-63D2153A0D1311D7AD06003065D5E7E4@[10.0.0.23]>

Greg Ward wrote:

> Just, can you run the following commands and tell me what you get:

I get this in all four cases:

['0x9', '0xa', '0xb', '0xc', '0xd', '0x20']

But: I can't reproduce the problem with -c, only in an interactive shell or in a
script! Here's what I get in an interactive session:

  sh-2.05a$ ./python.exe -S
  Python 2.3a0 (#42, Dec 11 2002, 10:33:42) 
  [GCC 3.1 20020420 (prerelease)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import string; print [hex(ord(c)) for c in string.whitespace]
  ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20', '0xa0']
  >>> 

(Look, now there's an 0xa0 in there!)

  sh-2.05a$ ./python.exe 
  Python 2.3a0 (#42, Dec 11 2002, 10:33:42) 
  [GCC 3.1 20020420 (prerelease)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import string; print [hex(ord(c)) for c in string.whitespace]
  ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20']
  >>> 

  sh-2.05a$ LANG=C ./python.exe -S
  Python 2.3a0 (#42, Dec 11 2002, 10:33:42) 
  [GCC 3.1 20020420 (prerelease)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import string; print [hex(ord(c)) for c in string.whitespace]
  ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20']
  >>> 

  sh-2.05a$ LANG=C ./python.exe   
  Python 2.3a0 (#42, Dec 11 2002, 10:33:42) 
  [GCC 3.1 20020420 (prerelease)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import string; print [hex(ord(c)) for c in string.whitespace]
  ['0x9', '0xa', '0xb', '0xc', '0xd', '0x20']
  >>> 

I have _no_ idea what's going on here.

Just


From martin@v.loewis.de  Wed Dec 11 14:23:10 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 15:23:10 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <017b01c2a11e$4b655c60$0900a8c0@spiff>
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net>
 <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]>
 <20021211134958.GA10384@cthulhu.gerg.ca>
 <017b01c2a11e$4b655c60$0900a8c0@spiff>
Message-ID: <m37kegfxdd.fsf@mira.informatik.hu-berlin.de>

"Fredrik Lundh" <fredrik@pythonware.com> writes:

> > Weird.  What the heck is 0xa0 doing in string.whitespace?  Oh, waitasec,
> > 0xa0 is non-breaking-space in ISO-8859-1.  But no, string.whitespace is
> > hardcoded to the US-ASCII whitespace chars, all of which are in
> > range(128) -- your locale shouldn't matter.
[...]
>     /* Create 'whitespace' object */
>     n = 0;
>     for (c = 0; c < 256; c++) {
>         if (isspace(c))
>             buf[n++] = c;
>     }

Sure. But shouldn't Python/the-C-library startup in the "C" locale?

Is this another one of these Mac quirks?

Regards,
Martin


From mwh@python.net  Wed Dec 11 14:38:20 2002
From: mwh@python.net (Michael Hudson)
Date: 11 Dec 2002 14:38:20 +0000
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: Just van Rossum's message of "Wed, 11 Dec 2002 15:17:59 +0100"
References: <r01050400-1022-63D2153A0D1311D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <2mel8owrhf.fsf@starship.python.net>

Just van Rossum <just@letterror.com> writes:

> I have _no_ idea what's going on here.

I do: readline.

I wouldn't want to live without readline, but some of the things it
does call for the application of thumbscrews.

Cheers,
M.

-- 
    FORD:  Just pust the fish in your ear, come on, it's only a
           little one.
  ARTHUR:  Uuuuuuuuggh!
                    -- The Hitch-Hikers Guide to the Galaxy, Episode 1


From gward@python.net  Wed Dec 11 14:39:57 2002
From: gward@python.net (Greg Ward)
Date: Wed, 11 Dec 2002 09:39:57 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <017b01c2a11e$4b655c60$0900a8c0@spiff>
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net> <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]> <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff>
Message-ID: <20021211143957.GA10597@cthulhu.gerg.ca>

[/F proves beyond a shadow of a doubt that string.whitespace is
 locale-sensitive]

Thanks, Fredrik!  That clarifies the behaviour Just is seeing.

Hey: I just realized that making textwrap trust string.whitespace is
wrong in at least one case: 0xa0 is *non-breaking* space in ISO-8859-1,
and converting it to 0x20 (regular ol' space) is clearly wrong -- the
"non-break" request will be ignored.  So Unicode or not, textwrap should
probably just hard-code the US-ASCII whitespace chars.

My attitude is that textwrap should work on European languages, whether
they are encoded in 8-bit "ASCII" or Unicode.  I suspect that passing an
arbitrary Unicode string to it is meaningles -- what the heck does it
even mean to wrap a string of Chinese or Hebrew or Devangari characters?
Beats me, and I think they're out of scope for textwrap.

So: do I even need to worry about the cornucopia of Unicode whitespace
characters at all?  Or can I sweep that can of worms under the rug?
(Pardon the horribly mixed metaphor.)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Do radioactive cats have 18 half-lives?


From guido@python.org  Wed Dec 11 15:25:41 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 11 Dec 2002 10:25:41 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: Your message of "Wed, 11 Dec 2002 08:49:58 EST."
 <20021211134958.GA10384@cthulhu.gerg.ca>
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net> <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]>
 <20021211134958.GA10384@cthulhu.gerg.ca>
Message-ID: <200212111525.gBBFPfh09916@pcp02138704pcs.reston01.va.comcast.net>

> Weird.  What the heck is 0xa0 doing in string.whitespace?  Oh,
> waitasec, 0xa0 is non-breaking-space in ISO-8859-1.  But no,
> string.whitespace is hardcoded to the US-ASCII whitespace chars, all
> of which are in range(128) -- your locale shouldn't matter.

It's overridden in the strop module, which tests for isspace().

--Guido van Rossum (home page: http://www.python.org/~guido/)


From martin@v.loewis.de  Wed Dec 11 15:27:09 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 16:27:09 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <r01050400-1022-63D2153A0D1311D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-63D2153A0D1311D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <m3lm2wefua.fsf@mira.informatik.hu-berlin.de>

Just van Rossum <just@letterror.com> writes:

> I have _no_ idea what's going on here.

Are you using readline in interactive mode? Do you have the current
CVS? readline sets LC_CTYPE, but the current CVS is supposed to
restore it to its original value.

Regards,
Martin



From guido@python.org  Wed Dec 11 15:37:09 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 11 Dec 2002 10:37:09 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: Your message of "11 Dec 2002 14:38:20 GMT."
 <2mel8owrhf.fsf@starship.python.net>
References: <r01050400-1022-63D2153A0D1311D7AD06003065D5E7E4@[10.0.0.23]>
 <2mel8owrhf.fsf@starship.python.net>
Message-ID: <200212111537.gBBFb9v16268@pcp02138704pcs.reston01.va.comcast.net>

> I wouldn't want to live without readline, but some of the things it
> does call for the application of thumbscrews.

Indeed.

But I thought I fixed that in 2.3, by restoring the locale after
initializing GNU readline.  Maybe the Mac IDE environment starts up in
a different locale?  Would be typical for Mac...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Wed Dec 11 15:39:56 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 11 Dec 2002 10:39:56 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: Your message of "Wed, 11 Dec 2002 09:39:57 EST."
 <20021211143957.GA10597@cthulhu.gerg.ca>
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net> <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]> <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff>
 <20021211143957.GA10597@cthulhu.gerg.ca>
Message-ID: <200212111539.gBBFduf16609@pcp02138704pcs.reston01.va.comcast.net>

> [/F proves beyond a shadow of a doubt that string.whitespace is
>  locale-sensitive]
> 
> Thanks, Fredrik!  That clarifies the behaviour Just is seeing.
> 
> Hey: I just realized that making textwrap trust string.whitespace is
> wrong in at least one case: 0xa0 is *non-breaking* space in ISO-8859-1,
> and converting it to 0x20 (regular ol' space) is clearly wrong -- the
> "non-break" request will be ignored.  So Unicode or not, textwrap should
> probably just hard-code the US-ASCII whitespace chars.

+1

> My attitude is that textwrap should work on European languages, whether
> they are encoded in 8-bit "ASCII" or Unicode.  I suspect that passing an
> arbitrary Unicode string to it is meaningles -- what the heck does it
> even mean to wrap a string of Chinese or Hebrew or Devangari characters?
> Beats me, and I think they're out of scope for textwrap.

Correct -- you can't trust the width of characters to be all the
same.  (I'm not even sure if that's true for Latin-1, Cyrillic or
Greek, but it seems likely.)

> So: do I even need to worry about the cornucopia of Unicode whitespace
> characters at all?  Or can I sweep that can of worms under the rug?
> (Pardon the horribly mixed metaphor.)

Please shove them under the garage.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Wed Dec 11 15:45:31 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 11 Dec 2002 16:45:31 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <200212111537.gBBFb9v16268@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-9A94A0400D1F11D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> But I thought I fixed that in 2.3, by restoring the locale after
> initializing GNU readline.  Maybe the Mac IDE environment starts up in
> a different locale?  Would be typical for Mac...

I thought I clearly showed command line examples. This is a vanilla Unix Python.

Martin: yes, I've got readline installed and yes Python was built from CVS.

Just


From guido@python.org  Wed Dec 11 16:01:23 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 11 Dec 2002 11:01:23 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: Your message of "Wed, 11 Dec 2002 16:45:31 +0100."
 <r01050400-1022-9A94A0400D1F11D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-9A94A0400D1F11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212111601.gBBG1Nh16796@pcp02138704pcs.reston01.va.comcast.net>

> > But I thought I fixed that in 2.3, by restoring the locale after
> > initializing GNU readline.  Maybe the Mac IDE environment starts up in
> > a different locale?  Would be typical for Mac...
> 
> I thought I clearly showed command line examples. This is a vanilla
> Unix Python.
> 
> Martin: yes, I've got readline installed and yes Python was built from CVS.

Hm.  Mayb the code in Modules/readline.c somehow has the setlocale()
call #ifdef'ed out?  Can you put a printf inside the #ifdef
SAVE_LOCALE block to see if it is executed?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mal@lemburg.com  Wed Dec 11 16:36:25 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 11 Dec 2002 17:36:25 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39
 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
References: <E18M7Tw-0004CG-00@sc8-pr-cvs1.sourceforge.net>
Message-ID: <3DF76989.5070700@lemburg.com>

loewis@users.sourceforge.net wrote:
> Update of /cvsroot/python/python/dist/src/Include
> In directory sc8-pr-cvs1:/tmp/cvs-serv15966/Include
> 
> Modified Files:
> 	compile.h parsetok.h pyerrors.h pythonrun.h symtable.h 
> Log Message:
> Constify filenames and scripts. Fixes #651362.

I remember Tim opposing such changes (and Greg Stein insisting
on them ;-). AFAIK, this breaks third party code since some
compilers simply reject casting char* to const char*.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From guido@python.org  Wed Dec 11 16:43:59 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 11 Dec 2002 11:43:59 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56 symtable.h,2.10,2.11
In-Reply-To: Your message of "Wed, 11 Dec 2002 17:36:25 +0100."
 <3DF76989.5070700@lemburg.com>
References: <E18M7Tw-0004CG-00@sc8-pr-cvs1.sourceforge.net>
 <3DF76989.5070700@lemburg.com>
Message-ID: <200212111643.gBBGhxL17148@pcp02138704pcs.reston01.va.comcast.net>

> I remember Tim opposing such changes (and Greg Stein insisting
> on them ;-). AFAIK, this breaks third party code since some
> compilers simply reject casting char* to const char*.

That was years ago.  It also sounds like a standard-defying compiler.

If this is still an issue, we'll find out during the alpha or beta
testing and we can revert if really necessary.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From martin@v.loewis.de  Wed Dec 11 16:53:56 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 17:53:56 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <20021211143957.GA10597@cthulhu.gerg.ca>
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net>
 <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]>
 <20021211134958.GA10384@cthulhu.gerg.ca>
 <017b01c2a11e$4b655c60$0900a8c0@spiff>
 <20021211143957.GA10597@cthulhu.gerg.ca>
Message-ID: <m3y96wcx97.fsf@mira.informatik.hu-berlin.de>

Greg Ward <gward@python.net> writes:

> My attitude is that textwrap should work on European languages, whether
> they are encoded in 8-bit "ASCII" or Unicode.  

Please, don't assume any specific encoding. Why is Latin-1 better than
KOI8-R? The only encoding that is truly better than all others is
ASCII, since virtually all other encodings have ASCII as a subset
(except for the EBCDIC ones, and, with limitations, the ISO-2022
ones).

Also, you'll find more-and-more European languages encoded in UTF-8,
so your support would be useless and give wrong results.

[If you meant to suggest no specific processing for &nbsp; disregard
this comment]

> I suspect that passing an arbitrary Unicode string to it is
> meaningles -- what the heck does it even mean to wrap a string of
> Chinese or Hebrew or Devangari characters?  Beats me, and I think
> they're out of scope for textwrap.

Actually, the Unicode database has "line-breaking properties". Those
are not yet incorporated into unicodedata, but that could be used to
meaningfully extend the module to Unicode.

> So: do I even need to worry about the cornucopia of Unicode whitespace
> characters at all?  Or can I sweep that can of worms under the rug?
> (Pardon the horribly mixed metaphor.)

Sweep away.

Regards,
Martin


From martin@v.loewis.de  Wed Dec 11 16:56:16 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 17:56:16 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <200212111601.gBBG1Nh16796@pcp02138704pcs.reston01.va.comcast.net>
References: <r01050400-1022-9A94A0400D1F11D7AD06003065D5E7E4@[10.0.0.23]>
 <200212111601.gBBG1Nh16796@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <m3u1hkcx5b.fsf@mira.informatik.hu-berlin.de>

Guido van Rossum <guido@python.org> writes:

> > Martin: yes, I've got readline installed and yes Python was built from CVS.
> 
> Hm.  Mayb the code in Modules/readline.c somehow has the setlocale()
> call #ifdef'ed out?  Can you put a printf inside the #ifdef
> SAVE_LOCALE block to see if it is executed?

I'm confused by Just's observation that it happens *only* with -S, in
interactive mode. I would have expected that "plain" interactive mode
behaves the same, but this is not what Just reported.

Regards,
Martin



From tim@zope.com  Wed Dec 11 17:01:32 2002
From: tim@zope.com (Tim Peters)
Date: Wed, 11 Dec 2002 12:01:32 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
In-Reply-To: <3DF76989.5070700@lemburg.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEALDCAB.tim@zope.com>

[M.-A. Lemburg, on constification]
> I remember Tim opposing such changes (and Greg Stein insisting
> on them ;-). AFAIK, this breaks third party code since some
> compilers simply reject casting char* to const char*.

A non-broken compiler always allowd passing a

    T

actual argument to a

    const T

formal argument, so I never object to those, and indeed I sneak const into
prototype arglists whenever it's convenient.

That's a different issue than making all of Python "const correct", which
has been a near boundlessly low bang-for-the-buck effort in every project
I've seen undertake it.  Try passing a

    const T*

actual argument to a

    T*

formal argument and a non-broken compiler will complain, and then either
casts have to proliferate, or consts have to proliferate, to stop the
errors.  Sometimes this is called "const poisoning", at least by objective
observers <wink>.



From martin@v.loewis.de  Wed Dec 11 17:17:01 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 18:17:01 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEALDCAB.tim@zope.com>
References: <LNBBLJKPBEHFEDALKOLCIEALDCAB.tim@zope.com>
Message-ID: <m3d6o8cw6q.fsf@mira.informatik.hu-berlin.de>

"Tim Peters" <tim@zope.com> writes:

> Sometimes this is called "const poisoning", at least by objective
> observers <wink>.

For the specific change, I found quite a lot of const poisoning: in
the original bug report, the submitter only requested a single change
(for RunString). Changing the entire class of functions, and figuring
out the closure of needed changes, was my day's ratio of poison that I
can stand without becoming more insane :-)

I had to cut it short at two places: PyFile_FromFile got not changed,
and neither did the tok_state members.

Regards,
Martin



From dave@boost-consulting.com  Wed Dec 11 17:44:03 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Wed, 11 Dec 2002 12:44:03 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include
 compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64
 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEALDCAB.tim@zope.com> ("Tim Peters"'s
 message of "Wed, 11 Dec 2002 12:01:32 -0500")
References: <LNBBLJKPBEHFEDALKOLCIEALDCAB.tim@zope.com>
Message-ID: <u3cp4xxgc.fsf@boost-consulting.com>

"Tim Peters" <tim@zope.com> writes:

> A non-broken compiler always allowd passing a
>
>     T
>
> actual argument to a
>
>     const T
>
> formal argument, so I never object to those, and indeed I sneak const into
> prototype arglists whenever it's convenient.

I don't know if C differs from C++ in this respect, but in C++, const
in a prototype argument list has no effect whatsoever (unless
modifying the referent of a pointer or reference).  So:

     void f(const int x);

is identical to

     void f(int x);

In fact, a very useful idiom is:

     void f(int x); // prototype. No const; it communicates nothing
                     // useful to the caller

     ...

     void f(const int x)  // implementation. I know I can always use x
     {                    // and get the original argument value
        ...
     }

> That's a different issue than making all of Python "const correct", which
> has been a near boundlessly low bang-for-the-buck effort in every project
> I've seen undertake it.  Try passing a
>
>     const T*
>
> actual argument to a
>
>     T*
>
> formal argument and a non-broken compiler will complain, and then
> either casts have to proliferate, or consts have to proliferate, to
> stop the errors.  Sometimes this is called "const poisoning", at
> least by objective observers <wink>.

There's a long discussion even now on comp.lang.c++.moderated about
whether const-correctness is a dogma.

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From just@letterror.com  Wed Dec 11 17:54:40 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 11 Dec 2002 18:54:40 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <m3u1hkcx5b.fsf@mira.informatik.hu-berlin.de>
Message-ID: <r01050400-1022-A1A6849A0D3111D7AD06003065D5E7E4@[10.0.0.23]>

Martin v. L=F6wis wrote:

> I'm confused by Just's observation that it happens *only* with -S, in
> interactive mode. I would have expected that "plain" interactive mode
> behaves the same, but this is not what Just reported.

One more time:

  sh-2.05a$ ./python.exe=20
  Python 2.3a0 (#42, Dec 11 2002, 10:33:42)=20
  [GCC 3.1 20020420 (prerelease)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import string; string.whitespace
  '\t\n\x0b\x0c\r '
  >>>=20

  sh-2.05a$ ./python.exe -S
  Python 2.3a0 (#42, Dec 11 2002, 10:33:42)=20
  [GCC 3.1 20020420 (prerelease)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import string; string.whitespace
  '\t\n\x0b\x0c\r \xa0'
  >>>=20

  sh-2.05a$ LANG=3DC ./python.exe -S
  Python 2.3a0 (#42, Dec 11 2002, 10:33:42)=20
  [GCC 3.1 20020420 (prerelease)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import string; string.whitespace
  '\t\n\x0b\x0c\r '
  >>>=20

Guys, I don't care about enough about this problem (if it is one) to dig =
any
deeper. I just happened to notice the textwrap breakage when I was import=
ing
lots of arbitrary modules to do some import timing...

Just


From barry@zope.com  Wed Dec 11 18:12:55 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Wed, 11 Dec 2002 13:12:55 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include
 compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64
 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
References: <LNBBLJKPBEHFEDALKOLCIEALDCAB.tim@zope.com>
 <u3cp4xxgc.fsf@boost-consulting.com>
Message-ID: <15863.32807.655604.178384@gargle.gargle.HOWL>

>>>>> "DA" == David Abrahams <dave@boost-consulting.com> writes:

    DA> There's a long discussion even now on comp.lang.c++.moderated
    DA> about whether const-correctness is a dogma.

Wow, and you think some Python-related threads never die.

-Barry


From martin@v.loewis.de  Wed Dec 11 18:58:11 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: Wed, 11 Dec 2002 19:58:11 +0100
Subject: [Python-Dev] Modules/tclNotify.c
Message-ID: <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de>

Is this file still in use? I stumbled over it while investigating
support for threaded Tcl.

If it is not used anymore, can it be deleted?

Regards,
Martin



From guido@python.org  Wed Dec 11 19:07:27 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 11 Dec 2002 14:07:27 -0500
Subject: [Python-Dev] Modules/tclNotify.c
In-Reply-To: Your message of "Wed, 11 Dec 2002 19:58:11 +0100."
 <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de>
References: <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de>
Message-ID: <200212111907.gBBJ7RI18188@pcp02138704pcs.reston01.va.comcast.net>

> Is this file still in use? I stumbled over it while investigating
> support for threaded Tcl.
> 
> If it is not used anymore, can it be deleted?

It was my experiment.  I don't rememeber what came of it; I guess it
wasn't needed.  You should also delete the 'license.terms' if you
delete tclNotify.c.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim@zope.com  Wed Dec 11 19:16:56 2002
From: tim@zope.com (Tim Peters)
Date: Wed, 11 Dec 2002 14:16:56 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
In-Reply-To: <u3cp4xxgc.fsf@boost-consulting.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEBLDCAB.tim@zope.com>

[Tim]
>> A non-broken compiler always allowd passing a
>>
>>     T
>>
>> actual argument to a
>>
>>     const T
>>
>> ...

[David Abrahams]
> I don't know if C differs from C++ in this respect, but in C++, const
> in a prototype argument list has no effect whatsoever (unless
> modifying the referent of a pointer or reference).

I meant T above to include cases like T = char*.

>  So:
>
>      void f(const int x);
>
> is identical to
>
>      void f(int x);

Not in the presence of

#define int char*

it's not.

> In fact, a very useful idiom is:
>
>      void f(int x); // prototype. No const; it communicates nothing
>                      // useful to the caller
>
>      ...
>
>      void f(const int x)  // implementation. I know I can always use x
>      {                    // and get the original argument value
>         ...
>      }

All seriousness aside, that's useful in C too, but rarely seen, since the
meaning of C code is always self-evident.

> ...
> There's a long discussion even now on comp.lang.c++.moderated about
> whether const-correctness is a dogma.

I prefer to think of it as a crock, except when I'm working on embedded C
applications that need to know which static data is ROMable.  So far that
hasn't happened.  We came close once at Dragon, until the vendor revealed
their compiler didn't support static data at all.

That said, I'm happy to add const decorations to Python's C API decls where
it helps C++ users, provided we get to stop about a thousand miles short of
making anyone truly happy.

compromise-is-the-art-of-spreading-misery-ly y'rs  - tim



From martin@v.loewis.de  Wed Dec 11 19:54:07 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 20:54:07 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEBLDCAB.tim@zope.com>
References: <LNBBLJKPBEHFEDALKOLCIEBLDCAB.tim@zope.com>
Message-ID: <m3vg20bacg.fsf@mira.informatik.hu-berlin.de>

"Tim Peters" <tim@zope.com> writes:

> >> A non-broken compiler always allowd passing a
> >>
> >>     T
> >>
> >> actual argument to a
> >>
> >>     const T
> >>
> >> ...
> 
> I meant T above to include cases like T = char*.

Of course, if T = char*, then const T = char *const != const char*.

Regards,
Martin


From martin@v.loewis.de  Wed Dec 11 19:56:15 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 20:56:15 +0100
Subject: [Python-Dev] Modules/tclNotify.c
In-Reply-To: <200212111907.gBBJ7RI18188@pcp02138704pcs.reston01.va.comcast.net>
References: <200212111858.gBBIwB7Z023487@mira.informatik.hu-berlin.de>
 <200212111907.gBBJ7RI18188@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <m3r8coba8w.fsf@mira.informatik.hu-berlin.de>

Guido van Rossum <guido@python.org> writes:

> It was my experiment.  I don't rememeber what came of it; I guess it
> wasn't needed.  You should also delete the 'license.terms' if you
> delete tclNotify.c.

Thanks for the information! I've now removed both files.

Regards,
Martin


From dave@boost-consulting.com  Wed Dec 11 20:44:39 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Wed, 11 Dec 2002 15:44:39 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include
 compile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64
 pythonrun.h,2.55,2.56	symtable.h,2.10,2.11
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEBLDCAB.tim@zope.com> ("Tim Peters"'s
 message of "Wed, 11 Dec 2002 14:16:56 -0500")
References: <LNBBLJKPBEHFEDALKOLCIEBLDCAB.tim@zope.com>
Message-ID: <uznrcl1zc.fsf@boost-consulting.com>

"Tim Peters" <tim@zope.com> writes:

> [Tim]
>>> A non-broken compiler always allowd passing a
>>> T
>>> actual argument to a
>>> const T
>>> ...
>
> [David Abrahams]
>> I don't know if C differs from C++ in this respect, but in C++,
>> const in a prototype argument list has no effect whatsoever (unless
>> modifying the referent of a pointer or reference).
>
> I meant T above to include cases like T = char*.
>
>>  So:
>> void f(const int x);
>> is identical to
>> void f(int x);
>
> Not in the presence of
>
> #define int char*
>
> it's not.

Ouch. Now "int const" != "const int" (and I seriously can't think of
any other problems with that #define. Macros to the rescue!!!
... uh, <wink>)

> All seriousness aside, that's useful in C too, but rarely seen, since
> the meaning of C code is always self-evident.

Better than that. It always works perfectly and is trivial to write
correctly.  Who needs Python when we have C?

> That said, I'm happy to add const decorations to Python's C API decls
> where it helps C++ users, provided we get to stop about a thousand
> miles short of making anyone truly happy.

That sure would reduce the number of nasty casts we need to make.
const_casts always set C++ users' teeth to chatterin'.

> compromise-is-the-art-of-spreading-misery-ly y'rs - tim

...somewhat.

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From tdelaney@avaya.com  Wed Dec 11 21:45:19 2002
From: tdelaney@avaya.com (Delaney, Timothy)
Date: Thu, 12 Dec 2002 08:45:19 +1100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co
 mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2
 .55,2.56	symtable.h,2.10,2.11
Message-ID: <B43D149A9AB2D411971300B0D03D7E8BF0A7C1@natasha.auslabs.avaya.com>

> From: M.-A. Lemburg [mailto:mal@lemburg.com]
> 
> I remember Tim opposing such changes (and Greg Stein insisting
> on them ;-). AFAIK, this breaks third party code since some
> compilers simply reject casting char* to const char*.

I must admit I caused this recent constifying mess ... someone complained on
python-list and I told him to submit a bug report (in the hope that it would
get rejected because I knew the problems it would cause).

I submit myself for a spanking.

Tim Delaney


From martin@v.loewis.de  Wed Dec 11 22:06:27 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 23:06:27 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56	symtable.h,2.10,2.11
In-Reply-To: <B43D149A9AB2D411971300B0D03D7E8BF0A7C1@natasha.auslabs.avaya.com>
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C1@natasha.auslabs.avaya.com>
Message-ID: <m365u0b47w.fsf@mira.informatik.hu-berlin.de>

"Delaney, Timothy" <tdelaney@avaya.com> writes:

> I must admit I caused this recent constifying mess ... someone complained on
> python-list and I told him to submit a bug report (in the hope that it would
> get rejected because I knew the problems it would cause).

I (obviously) did not feel that the request was misguided. The second
on (on RunString) was probably more effort that it will ever do good,
but I am convinced that the change is conceptually right, i.e. the
strings used as file names and scripts really should be const, as
people will pass string literals.

OTOH, there are more places left that should accept const strings, but
don't, and I was certainly not out to fix them all. Instead, doing it
on user request only seems right to me. In the long run, we get happy
users because of such changes, which should be weighed against the few
unhappy developers that now have to silence compiler warnings :-)

Regards,
Martin



From mal@lemburg.com  Wed Dec 11 22:45:54 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 11 Dec 2002 23:45:54 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co
 mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2
 .55,2.56	symtable.h,2.10,2.11
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C1@natasha.auslabs.avaya.com> <m365u0b47w.fsf@mira.informatik.hu-berlin.de>
Message-ID: <3DF7C022.3060701@lemburg.com>

Martin v. L=F6wis wrote:
> "Delaney, Timothy" <tdelaney@avaya.com> writes:
>=20
>>I must admit I caused this recent constifying mess ... someone complain=
ed on
>>python-list and I told him to submit a bug report (in the hope that it =
would
>>get rejected because I knew the problems it would cause).
>=20
>=20
> I (obviously) did not feel that the request was misguided. The second
> on (on RunString) was probably more effort that it will ever do good,
> but I am convinced that the change is conceptually right, i.e. the
> strings used as file names and scripts really should be const, as
> people will pass string literals.
 >
> OTOH, there are more places left that should accept const strings, but
> don't, and I was certainly not out to fix them all. Instead, doing it
> on user request only seems right to me. In the long run, we get happy
> users because of such changes, which should be weighed against the few
> unhappy developers that now have to silence compiler warnings :-)

I would accept that point if you could demonstrate a single
case where the const nature of the filename actually does any
good for the *user*.

BTW, how can I silence warnings when writing C code that's
supposed to compiler with Python 2.1 and 2.3 ? (passing const char *
to a char * API doesn't work for obvious reasons)

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From mhammond@skippinet.com.au  Wed Dec 11 22:59:07 2002
From: mhammond@skippinet.com.au (Mark Hammond)
Date: Thu, 12 Dec 2002 09:59:07 +1100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56	symtable.h,2.10,2.11
In-Reply-To: <3DF7C022.3060701@lemburg.com>
Message-ID: <LCEPIIGDJPKCOIHOBJEPOENMICAA.mhammond@skippinet.com.au>

[Martin]
> > OTOH, there are more places left that should accept const strings, but
> > don't, and I was certainly not out to fix them all. Instead, doing it
> > on user request only seems right to me. In the long run, we get happy
> > users because of such changes, which should be weighed against the few
> > unhappy developers that now have to silence compiler warnings :-)

[MAL]
> I would accept that point if you could demonstrate a single
> case where the const nature of the filename actually does any
> good for the *user*.

Well, it depends how you define "user".  When working on win32all, I am a
developer.  However, when knocking up a simple little extension for Python
for some silly reason, I feel more like a user.

I'm with Martin on this on; these strings should be const.  It has bitten me
*lots* of times.  Sometimes as I am implementing someone elses API, am
passed a "const char *", but can't simply pass it to Python API functions
that clearly treat the string as const.  Similarly, I tend to personally use
const for my own APIs where possible, so often I do this just to find my
const-ness getting in the way.

I haven't been very vocal about it as I see both side of the const
argument - but personally lean towards the "const is good" camp.

> BTW, how can I silence warnings when writing C code that's
> supposed to compiler with Python 2.1 and 2.3 ? (passing const char *
> to a char * API doesn't work for obvious reasons)

This won't be a problem - continue to personally ignore the "const"
qualifier.  Then you are passing non-const strings to both the non-const 2.2
and the const 2.3.

It is only a problem if you want to take advantage of const - and if you
object in principle, you probably won't <wink>.  For the rest of us we just
gotta make the same decision we do these days about Python 1.5 etc - and
that probably means generally ignoring it for a few years too.

Mark.



From brett@python.org  Wed Dec 11 23:28:08 2002
From: brett@python.org (Brett Cannon)
Date: Wed, 11 Dec 2002 15:28:08 -0800 (PST)
Subject: [Python-Dev] Still want an expected skip file for regrtest.py?
Message-ID: <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU>

I just updated my CVS copy of Python and noticed that regrtest.py was
patched.  That triggered the memory of summarizing the whole expected skip
thread from the last summary.  The conclusion to that thread was the idea
of having a file that would list tests that could be added to  the
expected skip  list at run-time and thus  squash the issue of having to
ignore skipped test messages for things you knew would be skipped that
were not listed on the platform skip list.

Well, with ``dummy_thread(ing)?`` sitting there on SF and knowing that I
have a lot of PHP coding coming up, I am up for implementing this (with no
specified timeframe).  The question is how do you want it implemented?  I
say it should just be a file you keep in your Lib/test/ directory that is
formatted like what you pass in for the -f option to regrtest.py (heck, I
plan to copy the code that implements that option).  It should always be
included without a passed-in option since the whole point of this is that
these tests are expected to be skipped.  Thus it should have a
standardized name (how about "ToSkip.txt"? Maybe a leading underscore?)

The other issue is whether the listed tests should supplement or extend
the expected tests for the platform.  I say it should extend, but the
thread originally discussing this said supplement.  If both ways are
desired, the file could specify either ``..add`` or ``..replace`` (or
something similar) to signal which to do (default being to add them).  It
could be required to be at the top of the file, at the end, or just
anywhere; I have no preference.  But I think it should just extend it
since we have those tests listed for a reason and having to possibly
re-list them would be a pain.

Or you could all just say "surprise me, Brett" and let me sweat the
details.  =)

-Brett


From jeremy@alum.mit.edu  Wed Dec 11 23:32:42 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Wed, 11 Dec 2002 18:32:42 -0500
Subject: [Python-Dev] trace.py in std library?
Message-ID: <15863.51994.549143.421230@slothrop.zope.com>

There's a trace.py script sitting in Tools/scripts that works when
imported.  I used it today to add code coverage to Zope's test runner
script.  It took about 30 min. (mostly fixing bugs in trace.py) and
seems to work pretty well.

Why don't we put this script in the std library so it can actually be
imported?

I can guess at some of the reasons.  The code looks very complicated
and has been updated piecemeal by many people of the years.  The last
update has a bunch of comments with Zooko's name on them and a date.
They seem like XXX comments.

It might be nice to clean it up before putting in the library, but
I'd be in favor of having some minimally functional tool in the
library regardless.

Jeremy



From sholden@holdenweb.com  Wed Dec 11 23:47:09 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Wed, 11 Dec 2002 18:47:09 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56	symtable.h,2.10,2.11
References: <LCEPIIGDJPKCOIHOBJEPOENMICAA.mhammond@skippinet.com.au>
Message-ID: <005d01c2a16f$9ee64c90$6300000a@holdenweb.com>

> [Martin]
> > > OTOH, there are more places left that should accept const strings, but
> > > don't, and I was certainly not out to fix them all. Instead, doing it
> > > on user request only seems right to me. In the long run, we get happy
> > > users because of such changes, which should be weighed against the few
> > > unhappy developers that now have to silence compiler warnings :-)
>
> [MAL]
> > I would accept that point if you could demonstrate a single
> > case where the const nature of the filename actually does any
> > good for the *user*.
>
> Well, it depends how you define "user".  When working on win32all, I am a
> developer.  However, when knocking up a simple little extension for Python
> for some silly reason, I feel more like a user.
>
> I'm with Martin on this on; these strings should be const.  It has bitten
me
> *lots* of times.  Sometimes as I am implementing someone elses API, am
> passed a "const char *", but can't simply pass it to Python API functions
> that clearly treat the string as const.  Similarly, I tend to personally
use
> const for my own APIs where possible, so often I do this just to find my
> const-ness getting in the way.
>
> I haven't been very vocal about it as I see both side of the const
> argument - but personally lean towards the "const is good" camp.
>
> > BTW, how can I silence warnings when writing C code that's
> > supposed to compiler with Python 2.1 and 2.3 ? (passing const char *
> > to a char * API doesn't work for obvious reasons)
>
> This won't be a problem - continue to personally ignore the "const"
> qualifier.  Then you are passing non-const strings to both the non-const
2.2
> and the const 2.3.
>
> It is only a problem if you want to take advantage of const - and if you
> object in principle, you probably won't <wink>.  For the rest of us we
just
> gotta make the same decision we do these days about Python 1.5 etc - and
> that probably means generally ignoring it for a few years too.
>
> Mark.

Folks: I'm just reposting the latest contribution to this thread in an
attempt to stop the messages from appearing as attachments. Don't know
whether it will work, or why it was happening in the first place

-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Previous .sig file retired to                    www.homeforoldsigs.com
-----------------------------------------------------------------------




From martin@v.loewis.de  Wed Dec 11 22:51:12 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 11 Dec 2002 23:51:12 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56	symtable.h,2.10,2.11
In-Reply-To: <3DF7C022.3060701@lemburg.com>
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C1@natasha.auslabs.avaya.com>
 <m365u0b47w.fsf@mira.informatik.hu-berlin.de>
 <3DF7C022.3060701@lemburg.com>
Message-ID: <m3wumg9nkv.fsf@mira.informatik.hu-berlin.de>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> I would accept that point if you could demonstrate a single
> case where the const nature of the filename actually does any
> good for the *user*.

See the original bug report. It avoids having to write const_casts.

> BTW, how can I silence warnings when writing C code that's
> supposed to compiler with Python 2.1 and 2.3 ? (passing const char *
> to a char * API doesn't work for obvious reasons)

Can you explain where such silencing would be necessary, for the
specific patch in question? It is not, because non of the variables
that got const now are ever passed to 2.1 API: they are all internal
to the core.

Regards,
Martin



From barry@python.org  Wed Dec 11 23:54:11 2002
From: barry@python.org (Barry A. Warsaw)
Date: Wed, 11 Dec 2002 18:54:11 -0500
Subject: [Python-Dev] trace.py in std library?
References: <15863.51994.549143.421230@slothrop.zope.com>
Message-ID: <15863.53283.298026.362146@gargle.gargle.HOWL>

>>>>> "JH" == Jeremy Hylton <jeremy@alum.mit.edu> writes:

    JH> There's a trace.py script sitting in Tools/scripts that works
    JH> when imported.  I used it today to add code coverage to Zope's
    JH> test runner script.  It took about 30 min. (mostly fixing bugs
    JH> in trace.py) and seems to work pretty well.

Didn't Skip work on some code coverage tool?  If so, maybe he can tell
us how they compare.

-Barry


From jeremy@alum.mit.edu (Jeremy Hylton)  Wed Dec 11 23:55:41 2002
From: jeremy@alum.mit.edu (Jeremy Hylton) (Jeremy Hylton)
Date: Wed, 11 Dec 2002 18:55:41 -0500
Subject: [Python-Dev] trace.py in std library?
In-Reply-To: <15863.53283.298026.362146@gargle.gargle.HOWL>
References: <15863.51994.549143.421230@slothrop.zope.com>
 <15863.53283.298026.362146@gargle.gargle.HOWL>
Message-ID: <15863.53373.521756.396364@slothrop.zope.com>

trace.py is Skip's tool.

Jeremy



From tim.one@comcast.net  Wed Dec 11 23:59:23 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 11 Dec 2002 18:59:23 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co
 mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2
 .55,2.56	symtable.h,2.10,2.11
In-Reply-To: <005d01c2a16f$9ee64c90$6300000a@holdenweb.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEDKDCAB.tim.one@comcast.net>

> Folks: I'm just reposting the latest contribution to this thread in an
> attempt to stop the messages from appearing as attachments. Don't know
> whether it will work,

Sounds as likely to work as gutting a cat and offering the entrails to
Barry.

> or why it was happening in the first place

It wasn't -- I expect you're the only one who saw attachments.  My Outlook
2000 gets into that state sometimes, and is then cured by closing it, using
Task Manager to make sure it's gone, and then opening it again.



From barry@python.org  Thu Dec 12 00:10:09 2002
From: barry@python.org (Barry A. Warsaw)
Date: Wed, 11 Dec 2002 19:10:09 -0500
Subject: [Python-Dev] trace.py in std library?
References: <15863.51994.549143.421230@slothrop.zope.com>
 <15863.53283.298026.362146@gargle.gargle.HOWL>
 <15863.53373.521756.396364@slothrop.zope.com>
Message-ID: <15863.54241.591375.904547@gargle.gargle.HOWL>

>>>>> "JH" == Jeremy Hylton <jeremy@alum.mit.edu> writes:

    JH> trace.py is Skip's tool.

So I guess that means they're similar.

<wink>
-Barry


From barry@python.org  Thu Dec 12 00:12:24 2002
From: barry@python.org (Barry A. Warsaw)
Date: Wed, 11 Dec 2002 19:12:24 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co
 mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2
 .55,2.56	symtable.h,2.10,2.11
References: <005d01c2a16f$9ee64c90$6300000a@holdenweb.com>
 <LNBBLJKPBEHFEDALKOLCKEDKDCAB.tim.one@comcast.net>
Message-ID: <15863.54376.614967.808943@gargle.gargle.HOWL>

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

    >> Folks: I'm just reposting the latest contribution to this
    >> thread in an attempt to stop the messages from appearing as
    >> attachments. Don't know whether it will work,

    TP> Sounds as likely to work as gutting a cat and offering the
    TP> entrails to Barry.

I'm much more into fried snakes in my computer's guts:

http://www.uq.edu.au/education/extra/all.html

-Barry


From tdelaney@avaya.com  Thu Dec 12 00:34:16 2002
From: tdelaney@avaya.com (Delaney, Timothy)
Date: Thu, 12 Dec 2002 11:34:16 +1100
Subject: [Python-Dev] trace.py in std library?
Message-ID: <B43D149A9AB2D411971300B0D03D7E8BF0A7C5@natasha.auslabs.avaya.com>

> From: barry@python.org [mailto:barry@python.org]
> 
> >>>>> "JH" == Jeremy Hylton <jeremy@alum.mit.edu> writes:
> 
>     JH> trace.py is Skip's tool.
> 
> So I guess that means they're similar.

I think it's premature to put it into the std library - not least because
I've got a version I've been working on for a while which I'd eventually
like to get in there if I can ;) Of course, part of that will rest on my
employer ...

The major problems I had with trace.py are that it is *slow* and that the
data it collects is very incomplete.

If we are going to have a standard tracing module, I think it needs to be
*fast* and the data it collects needs to be complete, and easily
transformable to many types of reports. My main use case is automated code
coverage reports. If my unit tests take 30 minutes to run with no coverage
tool, but 15 hours to run under a coverage tool, the coverage tool is
useless for my nightly builds and automated testing.

The one I've been working on is a *lot* faster (though still in pure
python), dropping from about 3000% performance overhead to about 50%. It
also collects pretty much all the data it can (calls, lines, which lines
belong to which classes/functions/methods), resolves modules down to a
single, most-qualified instance (for example, if import a and import b.a end
up importing the same module, both will be represented in the report as b.a)
and various other things. It also determines which source lines are
executable (for example, multi-line strings will only have one of the lines
as executable).

It also *almost* works with Jython (doesn't work out executable lines
properly).

It *also* allows outputting of tracing lines as they are executed (and damn
that slows things down).

But I don't think it's a candidate yet. Among other things, the tracing
functionality is not hooked into each new thread. Not an issue for my
current project (which must run single-threaded due to being invoked inside
something that must run in a single thread) but it is an issue for a version
in the standard library.

Tim Delaney


From jeremy@alum.mit.edu  Thu Dec 12 00:38:19 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Wed, 11 Dec 2002 19:38:19 -0500
Subject: [Python-Dev] trace.py in std library?
In-Reply-To: <B43D149A9AB2D411971300B0D03D7E8BF0A7C5@natasha.auslabs.avaya.com>
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C5@natasha.auslabs.avaya.com>
Message-ID: <15863.55931.936197.13180@slothrop.zope.com>

>>>>> "TD" == Timothy Delaney <Delaney> writes:

  TD> But I don't think it's a candidate yet. Among other things, the
  TD> tracing functionality is not hooked into each new thread. Not an
  TD> issue for my current project (which must run single-threaded due
  TD> to being invoked inside something that must run in a single
  TD> thread) but it is an issue for a version in the standard
  TD> library.

Maybe you should checkin what you have now.  If it's at least as
functional as trace.py, then it's an improvement over what we've got
in the std library now.  You/we can improve it as 2.3 development
proceeds. 

Jeremy



From tdelaney@avaya.com  Thu Dec 12 00:46:14 2002
From: tdelaney@avaya.com (Delaney, Timothy)
Date: Thu, 12 Dec 2002 11:46:14 +1100
Subject: [Python-Dev] trace.py in std library?
Message-ID: <B43D149A9AB2D411971300B0D03D7E8BF0A7C7@natasha.auslabs.avaya.com>

> From: Jeremy Hylton [mailto:jeremy@alum.mit.edu]
> 
> >>>>> "TD" == Timothy Delaney <Delaney> writes:
> 
>   TD> But I don't think it's a candidate yet. Among other things, the
>   TD> tracing functionality is not hooked into each new thread. Not an
>   TD> issue for my current project (which must run single-threaded due
>   TD> to being invoked inside something that must run in a single
>   TD> thread) but it is an issue for a version in the standard
>   TD> library.
> 
> Maybe you should checkin what you have now.  If it's at least as
> functional as trace.py, then it's an improvement over what we've got
> in the std library now.  You/we can improve it as 2.3 development
> proceeds. 

Ooh - believe me - you don't want to see the code as it is now. Yuk! Needs
some major cleanup (esp. after the hacking I've done to make it work with
Jython). No way I would check something that disgusting in where other
people could see it ;)

Unfortunately, I'm really busy right now. Maybe I'll get to spend some time
on it over my 2-week break coming up.

Plus as I said, I need to check with my employer - it has been partially
developed on company time, and for a company project. Maybe I can get it
into my goals and objectives ;)

I should use the sandbox though. Unfortunately, I can't compile 2.3 at home
... MinGW still doesn't work, and I don't have a BSD or Linux box set up
(the only available machine is currently serving as my internet gateway
until I fix another machine).

Tim Delaney


From skip@pobox.com  Thu Dec 12 01:33:09 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 11 Dec 2002 19:33:09 -0600
Subject: [Python-Dev] trace.py in std library?
In-Reply-To: <15863.55931.936197.13180@slothrop.zope.com>
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C5@natasha.auslabs.avaya.com>
 <15863.55931.936197.13180@slothrop.zope.com>
Message-ID: <15863.59221.238508.808135@montanaro.dyndns.org>

    TD> But I don't think it's a candidate yet....

    Jeremy> Maybe you should checkin what you have now.  If it's at least as
    Jeremy> functional as trace.py, then it's an improvement over what we've
    Jeremy> got in the std library now.  You/we can improve it as 2.3
    Jeremy> development proceeds.

As the purported author of trace.py, I agree.  

Skip


From skip@pobox.com  Thu Dec 12 01:35:21 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 11 Dec 2002 19:35:21 -0600
Subject: [Python-Dev] trace.py in std library?
In-Reply-To: <B43D149A9AB2D411971300B0D03D7E8BF0A7C7@natasha.auslabs.avaya.com>
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C7@natasha.auslabs.avaya.com>
Message-ID: <15863.59353.890614.697677@montanaro.dyndns.org>

    Jeremy> Maybe you should checkin what you have now.

    Tim> Ooh - believe me - you don't want to see the code as it is
    Tim> now.

So check it into the sandbox (well, after your boss says okay).

Failing approval from your employer, can you outline the approach you took,
especially how it differs from the approach taken by trace.py?

Skip


From prabhu@aero.iitm.ernet.in  Thu Dec 12 04:10:44 2002
From: prabhu@aero.iitm.ernet.in (Prabhu Ramachandran)
Date: Thu, 12 Dec 2002 09:40:44 +0530
Subject: [Python-Dev] trace.py in std library?
In-Reply-To: <15863.51994.549143.421230@slothrop.zope.com>
References: <15863.51994.549143.421230@slothrop.zope.com>
Message-ID: <15864.3140.97153.574886@monster.linux.in>

>>>>> "JH" == Jeremy Hylton <jeremy@alum.mit.edu> writes:

    JH> There's a trace.py script sitting in Tools/scripts that works
    JH> when imported.  I used it today to add code coverage to Zope's
    JH> test runner script.  It took about 30 min. (mostly fixing bugs
    JH> in trace.py) and seems to work pretty well.

    JH> Why don't we put this script in the std library so it can
    JH> actually be imported?

FWIW I wrote a yet another trace.py script that I occasionally use to
look at call graphs.  Its pretty simple code and I have no idea how it
compares to Skip's version.  I do believe that I wrote to Skip about
my code a long while ago but can't remember what he said or if he said
anything at all.  My version pretty prints the call graph and indents
things to make things easier to read.  There is no 'coverage' class or
anything that gives you stats.  There are a few options which are
fairly well documented.  If you are interested maybe you can take a
look here:

 http://av.stanford.edu/~prabhu/download/python/

There is also a test suite for the code in that directory
(test_trace.py).

cheers,
prabhu


From mal@lemburg.com  Thu Dec 12 09:21:52 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 10:21:52 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co
 mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2
 .55,2.56	symtable.h,2.10,2.11
References: <LCEPIIGDJPKCOIHOBJEPOENMICAA.mhammond@skippinet.com.au>
Message-ID: <3DF85530.7040203@lemburg.com>

Mark Hammond wrote:
> [Martin]
> 
>>>OTOH, there are more places left that should accept const strings, but
>>>don't, and I was certainly not out to fix them all. Instead, doing it
>>>on user request only seems right to me. In the long run, we get happy
>>>users because of such changes, which should be weighed against the few
>>>unhappy developers that now have to silence compiler warnings :-)
> 
> [MAL]
> 
>>I would accept that point if you could demonstrate a single
>>case where the const nature of the filename actually does any
>>good for the *user*.
> 
> Well, it depends how you define "user".  When working on win32all, I am a
> developer.  However, when knocking up a simple little extension for Python
> for some silly reason, I feel more like a user.

It is definitely useful for the developer and that's what
I wanted to hint at: the user doesn't notice any change by
introducing "const".

> I'm with Martin on this on; these strings should be const.  It has bitten me
> *lots* of times.  Sometimes as I am implementing someone elses API, am
> passed a "const char *", but can't simply pass it to Python API functions
> that clearly treat the string as const.  Similarly, I tend to personally use
> const for my own APIs where possible, so often I do this just to find my
> const-ness getting in the way.
> 
> I haven't been very vocal about it as I see both side of the const
> argument - but personally lean towards the "const is good" camp.

I'm all for using "const" in new APIs too (I've added it to most
Unicode APIs for that reason), what worries me is that existing APIs
get converted.

>>BTW, how can I silence warnings when writing C code that's
>>supposed to compiler with Python 2.1 and 2.3 ? (passing const char *
>>to a char * API doesn't work for obvious reasons)
> 
> This won't be a problem - continue to personally ignore the "const"
> qualifier.  Then you are passing non-const strings to both the non-const 2.2
> and the const 2.3.
 >
> It is only a problem if you want to take advantage of const - and if you
> object in principle, you probably won't <wink>.  For the rest of us we just
> gotta make the same decision we do these days about Python 1.5 etc - and
> that probably means generally ignoring it for a few years too.

Will do.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From mal@lemburg.com  Thu Dec 12 09:30:56 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 10:30:56 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co
 mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2
 .55,2.56	symtable.h,2.10,2.11
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C1@natasha.auslabs.avaya.com>	<m365u0b47w.fsf@mira.informatik.hu-berlin.de>	<3DF7C022.3060701@lemburg.com> <m3wumg9nkv.fsf@mira.informatik.hu-berlin.de>
Message-ID: <3DF85750.7080109@lemburg.com>

Martin v. L=F6wis wrote:
> "M.-A. Lemburg" <mal@lemburg.com> writes:
>=20
>=20
>>I would accept that point if you could demonstrate a single
>>case where the const nature of the filename actually does any
>>good for the *user*.
>=20
> See the original bug report. It avoids having to write const_casts.

Great and now those who want to avoid warnings have to
patch their old code :-(

>>BTW, how can I silence warnings when writing C code that's
>>supposed to compiler with Python 2.1 and 2.3 ? (passing const char *
>>to a char * API doesn't work for obvious reasons)
>=20
> Can you explain where such silencing would be necessary, for the
> specific patch in question? It is not, because non of the variables
> that got const now are ever passed to 2.1 API: they are all internal
> to the core.

They are ? The pythonrun.h changes were public in 2.1 as well
and they include some of the more common APIs like the
PyRun_*() APIs.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Thu Dec 12 09:06:41 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Thu, 12 Dec 2002 10:06:41 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56	symtable.h,2.10,2.11
References: <LCEPIIGDJPKCOIHOBJEPOENMICAA.mhammond@skippinet.com.au> <3DF85530.7040203@lemburg.com>
Message-ID: <000901c2a1bd$c9a638c0$741ee8d9@mira>

> > Well, it depends how you define "user".  When working on win32all, I
am a
> > developer.  However, when knocking up a simple little extension for
Python
> > for some silly reason, I feel more like a user.
>
> It is definitely useful for the developer and that's what
> I wanted to hint at: the user doesn't notice any change by
> introducing "const".

I see that my terminology was confusing: user: software author who uses
the Python header files. developer: software author who writes them.
Strictly speaking, computer users never "use" Python: they always use
some Python application. So the Python "users" are always software
authors. Of those, only the authors who write C extensions will notice a
change. Sorry for the confusion.

> I'm all for using "const" in new APIs too (I've added it to most
> Unicode APIs for that reason), what worries me is that existing APIs
> get converted.

No need to worry. Nothing can break because of that.

Regards,
Martin



From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Thu Dec 12 09:11:42 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Thu, 12 Dec 2002 10:11:42 +0100
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Include co mpile.h,2.38,2.39 parsetok.h,2.19,2.20 pyerrors.h,2.63,2.64 pythonrun.h,2 .55,2.56	symtable.h,2.10,2.11
References: <B43D149A9AB2D411971300B0D03D7E8BF0A7C1@natasha.auslabs.avaya.com>	<m365u0b47w.fsf@mira.informatik.hu-berlin.de>	<3DF7C022.3060701@lemburg.com> <m3wumg9nkv.fsf@mira.informatik.hu-berlin.de> <3DF85750.7080109@lemburg.com>
Message-ID: <001101c2a1be$7cfbcc50$741ee8d9@mira>

> Great and now those who want to avoid warnings have to
> patch their old code :-(

Why is that? Nobody has to change anything.

>> Can you explain where such silencing would be necessary, for the
>> specific patch in question? It is not, because non of the variables
>> that got const now are ever passed to 2.1 API: they are all internal
>> to the core.

> They are ? The pythonrun.h changes were public in 2.1 as well
> and they include some of the more common APIs like the
>PyRun_*() APIs.

Right. But those changes will no author of extension modules force to
change anything. The PyRun_ functions now expect const char*. Existing
code will pass char*. No problem, this is well-formed C. In addition,
people who want to pass const char* can now do so without using a cast,
which they previously couldn't. The only necessary subsequential change
is *inside* the PyRun_ functions, where the types of local variables had
to be changed, and the types of PyParse_ functions, etc.

Developers of the Python core have to keep this in mind when changing
the Python core. Nobody else has to change anything.

Regards,
Martin



From mal@lemburg.com  Thu Dec 12 10:35:12 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 11:35:12 +0100
Subject: [Python-Dev] make install fails with current Python CVS
Message-ID: <3DF86660.6030808@lemburg.com>

I've tried to run a "make install" with todays CVS version, but
this fails on my Linux box with the following error:

Compiling /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep263.py ...
Sorry: MemoryError:

Is that just my setup or reproducable on other machines as well ?
The above file looks pretty innocent to me:

#! -*- coding: koi8-r -*-
assert u"<russian for python>".encode("utf-8") == '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd'
assert u"\<russian p>".encode("utf-8") == '\\\xd0\x9f'

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From just@letterror.com  Thu Dec 12 10:42:56 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 12 Dec 2002 11:42:56 +0100
Subject: [Python-Dev] new import hooks & zip import
Message-ID: <r01050400-1022-7C66986A0DBE11D7AD06003065D5E7E4@[10.0.0.23]>

After incorporating some suggestions and fixes from Paul Moore and Neal Norwitz
and doing more testing myself, I've decided the importhook/zipimport patch is
mature enough to post a proper patch:

   http://www.python.org/sf/652586

I've renamed two of the three new sys items; the patch note explains.

Thorough review would be very welcome. Thanks!

Just


From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Thu Dec 12 10:44:17 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Thu, 12 Dec 2002 11:44:17 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>
Message-ID: <00bb01c2a1cb$6bd13160$741ee8d9@mira>

> I've tried to run a "make install" with todays CVS version, but
> this fails on my Linux box with the following error:
>
> Compiling
/home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep26
3.py ...
> Sorry: MemoryError:

That is typically an indication that the codec lookup has failed. Can
you try to lookup the koi8-r codec interactively?

Regards,
Martin



From mal@lemburg.com  Thu Dec 12 11:01:26 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 12:01:26 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira>
Message-ID: <3DF86C86.7000508@lemburg.com>

Martin v. L=F6wis wrote:
>>I've tried to run a "make install" with todays CVS version, but
>>this fails on my Linux box with the following error:
>>
>>Compiling
>=20
> /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep2=
6
> 3.py ...
>=20
>>Sorry: MemoryError:
>=20
>=20
> That is typically an indication that the codec lookup has failed. Can
> you try to lookup the koi8-r codec interactively?

Hmm, I get:

Python/Dev-Python> ./python
Python 2.3a0 (#1, Dec 12 2002, 11:48:24)
[GCC 2.95.3 20010315 (SuSE)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> unicode('x', 'koi8-r')
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
LookupError: no codec search functions registered: can't find encoding
 >>>

Looking at the output of ./python -vv it appears as if the struct
cannot be loaded (which is imported by codecs.py).

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From mwh@python.net  Thu Dec 12 11:03:31 2002
From: mwh@python.net (Michael Hudson)
Date: 12 Dec 2002 11:03:31 +0000
Subject: [Python-Dev] make install fails with current Python CVS
In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 11:35:12 +0100"
References: <3DF86660.6030808@lemburg.com>
Message-ID: <2msmx3o5x8.fsf@starship.python.net>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> I've tried to run a "make install" with todays CVS version, but
> this fails on my Linux box with the following error:
> 
> Compiling /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep263.py ...
> Sorry: MemoryError:
> 
> Is that just my setup or reproducable on other machines as well ?

I saw that too, but thought it must have been my fault.

> The above file looks pretty innocent to me:
> 
> #! -*- coding: koi8-r -*-
> assert u"<russian for python>".encode("utf-8") == '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd'
> assert u"\<russian p>".encode("utf-8") == '\\\xd0\x9f'

It might well be the only file that actually uses the pep263 stuff to
affect the interpretation of a unicode string.  That would seem to be
the place to start looking -- though it imports & compiles file when
run as part of the test suite...

Hmm...

>>> exec open("../Lib/test/test_pep263.py").read()

sigh, that was one easy guess.

Cheers,
M.

-- 
  Every now and then, Google doesn't throw up what I need so I start
  checking Altavista, Yahoo, etc.  In almost every single case, I am
  brutally reminded why I use Google in the first place.
                                                  -- John Riddoch, asr


From mwh@python.net  Thu Dec 12 11:07:11 2002
From: mwh@python.net (Michael Hudson)
Date: 12 Dec 2002 11:07:11 +0000
Subject: [Python-Dev] make install fails with current Python CVS
In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 12:01:26 +0100"
References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>
Message-ID: <2mpts7o5r4.fsf@starship.python.net>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> Martin v. L=F6wis wrote:
> >>I've tried to run a "make install" with todays CVS version, but
> >>this fails on my Linux box with the following error:
[...]
> Looking at the output of ./python -vv it appears as if the struct
> cannot be loaded (which is imported by codecs.py).

Is this an out-of-tree build?  I have a feeling you may be being
bitten by their current breakage.

Weird symptom, though.

Cheers,
M.

-- 
59. In English every word can be verbed. Would that it were so in
    our programming languages.
  -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html


From mal@lemburg.com  Thu Dec 12 11:10:46 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 12:10:46 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>
Message-ID: <3DF86EB6.1020200@lemburg.com>

M.-A. Lemburg wrote:
> Martin v. L=F6wis wrote:
>=20
>>> I've tried to run a "make install" with todays CVS version, but
>>> this fails on my Linux box with the following error:
>>>
>>> Compiling
>>
>>
>> /home/lemburg/projects/Python/Installation/lib/python2.3/test/test_pep=
26
>> 3.py ...
>>
>>> Sorry: MemoryError:

Could this error message be made more helpful :-)

>>
>> That is typically an indication that the codec lookup has failed. Can
>> you try to lookup the koi8-r codec interactively?
>=20
>=20
> Hmm, I get:
>=20
> Python/Dev-Python> ./python
> Python 2.3a0 (#1, Dec 12 2002, 11:48:24)
> [GCC 2.95.3 20010315 (SuSE)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> unicode('x', 'koi8-r')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> LookupError: no codec search functions registered: can't find encoding
>  >>>
>=20
> Looking at the output of ./python -vv it appears as if the struct
> cannot be loaded (which is imported by codecs.py).

Might help:
For some reason I don't see a build/ dir in the Python distribution
directory.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From mal@lemburg.com  Thu Dec 12 11:12:33 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 12:12:33 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net>
Message-ID: <3DF86F21.6070509@lemburg.com>

Michael Hudson wrote:
> "M.-A. Lemburg" <mal@lemburg.com> writes:
> 
> 
>>Martin v. L=F6wis wrote:
>>
>>>>I've tried to run a "make install" with todays CVS version, but
>>>>this fails on my Linux box with the following error:
> 
> [...]
> 
>>Looking at the output of ./python -vv it appears as if the struct
>>cannot be loaded (which is imported by codecs.py).
> 
> 
> Is this an out-of-tree build? 

This is a CVS checkout from today. I then tried the usual
configure; make install dance and the byte-code compile
stopped with the error I posted.

> I have a feeling you may be being bitten by their current breakage.

It's still broken ? I though you cleared that problem out
last week ?

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From mwh@python.net  Thu Dec 12 11:39:53 2002
From: mwh@python.net (Michael Hudson)
Date: 12 Dec 2002 11:39:53 +0000
Subject: [Python-Dev] make install fails with current Python CVS
In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 12:12:33 +0100"
References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com>
Message-ID: <2mn0nbo48m.fsf@starship.python.net>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> Michael Hudson wrote:
> > "M.-A. Lemburg" <mal@lemburg.com> writes:
> > 
> > 
> >>Martin v. L=F6wis wrote:
> >>
> >>>>I've tried to run a "make install" with todays CVS version, but
> >>>>this fails on my Linux box with the following error:
> > 
> > [...]
> > 
> >>Looking at the output of ./python -vv it appears as if the struct
> >>cannot be loaded (which is imported by codecs.py).
> > 
> > 
> > Is this an out-of-tree build? 
> 
> This is a CVS checkout from today. I then tried the usual
> configure; make install dance and the byte-code compile
> stopped with the error I posted.

Oh.  Then maybe it's something else.

> > I have a feeling you may be being bitten by their current breakage.
> 
> It's still broken ? I though you cleared that problem out
> last week ?

No, I think it's still broken.  I handed the problem off to AMK --
don't /think/ he's fixed it yet...

Cheers,
M.

-- 
  My hat is lined with tinfoil for protection in the unlikely event
  that the droid gets his PowerPoint presentation working.
                               -- Alan W. Frame, alt.sysadmin.recovery


From mal@lemburg.com  Thu Dec 12 12:21:48 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 13:21:48 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>	<2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com> <2mn0nbo48m.fsf@starship.python.net>
Message-ID: <3DF87F5C.7060504@lemburg.com>

> "M.-A. Lemburg" <mal@lemburg.com> writes:
>>>I've tried to run a "make install" with todays CVS version, but
>>>this fails on my Linux box with the following error:
 >
>Looking at the output of ./python -vv it appears as if the struct
>cannot be loaded (which is imported by codecs.py).

Ok, I've narrowed down the problem to the missing shared mods
struct and _codecs. It seems that the distutils setup.py is
not run before installing and compiling the .py files.

There are two options:
1. tweak the Makefile targets
2. make struct and _codecs built-in modules

I think the second option is more attractive.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Thu Dec 12 13:19:49 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Thu, 12 Dec 2002 14:19:49 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <3DF86EB6.1020200@lemburg.com>
Message-ID: <00e301c2a1e1$25fa9d50$741ee8d9@mira>

> >>> Sorry: MemoryError:
>
> Could this error message be made more helpful :-)

I don't know. I tried, and failed. I did not try too hard, though.

I just knew what the problem was as I investigated this a couple of
times before, to track it down to the codecs lookup facility not
working.

Regards,
Martin



From niemeyer@conectiva.com  Thu Dec 12 13:24:40 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 12 Dec 2002 11:24:40 -0200
Subject: [Python-Dev] Tarfile support
Message-ID: <20021212112440.A25044@ibook.distro.conectiva>

Hello!

Lars Gustabel has opened a patch (#651082) with the implementation of
Tarfile, including documentation and everything. I have already reviewed
and sent him some considerations which were fixed (besides the
constructor stuff we discussed here). It would be nice if other people
reviewed it as well.

Thank you!

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From skip@pobox.com  Thu Dec 12 14:40:58 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 12 Dec 2002 08:40:58 -0600
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: <r01050400-1022-7C66986A0DBE11D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-7C66986A0DBE11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <15864.40954.242269.589277@montanaro.dyndns.org>

    Just> Thorough review would be very welcome.

Scanning the patch summary I see:

    sys.path_hooks is a list of callable objects that take 
    a string as their only argument. A hook will be called 
    with a sys.path or pkg.__path__ item. It should return 
    an "importer" object (see below), or raise ImportError 
    or return None if it can't deal with the path item. By 
    default, sys.path_hooks only contains the zipimporter 
    type, if the zipimport module is available. 

Why have it raise ImportError or return None if it can't deal with the path
item?  Shouldn't there be only one way to report failure?  (Or do I misread
your explanation?)

Skip


From just@letterror.com  Thu Dec 12 14:53:05 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 12 Dec 2002 15:53:05 +0100
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: <15864.40954.242269.589277@montanaro.dyndns.org>
Message-ID: <r01050400-1022-6DE170D80DE111D7AD06003065D5E7E4@[10.0.0.23]>

Skip Montanaro wrote:

> Scanning the patch summary I see:
> 
>     sys.path_hooks is a list of callable objects that take 
>     a string as their only argument. A hook will be called 
>     with a sys.path or pkg.__path__ item. It should return 
>     an "importer" object (see below), or raise ImportError 
>     or return None if it can't deal with the path item. By 
>     default, sys.path_hooks only contains the zipimporter 
>     type, if the zipimport module is available. 
> 
> Why have it raise ImportError or return None if it can't deal with the path
> item?  Shouldn't there be only one way to report failure?  (Or do I misread
> your explanation?)

No, you read correctly... I'm not sure, I just thought if might be convenient to
also be able to return None. I'd be happy to stick with just one way, though
(which would be raising ImportError, as that's easiest from __init__ methods).

Just


From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Thu Dec 12 13:25:17 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Thu, 12 Dec 2002 14:25:17 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>	<2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com> <2mn0nbo48m.fsf@starship.python.net> <3DF87F5C.7060504@lemburg.com>
Message-ID: <00ef01c2a1e1$e998bd50$741ee8d9@mira>

> Ok, I've narrowed down the problem to the missing shared mods
> struct and _codecs. It seems that the distutils setup.py is
> not run before installing and compiling the .py files.
>
> There are two options:
> 1. tweak the Makefile targets
> 2. make struct and _codecs built-in modules
>
> I think the second option is more attractive.

There is a third option:

3. Document that the proper build procedure is "configure;make;make
install"

Actually, that is documented, so I'd vote "pilot error".

That said, I'm always in favour of making more modules builtin again, as
this simplifies the build process and may slightly improve performance
if the module is imported.

Regards,
Martin



From guido@python.org  Thu Dec 12 15:56:05 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 10:56:05 -0500
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: Your message of "Thu, 12 Dec 2002 15:53:05 +0100."
 <r01050400-1022-6DE170D80DE111D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-6DE170D80DE111D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212121556.gBCFu5105528@odiug.zope.com>

> Skip Montanaro wrote:
> 
> > Scanning the patch summary I see:
> > 
> >     sys.path_hooks is a list of callable objects that take 
> >     a string as their only argument. A hook will be called 
> >     with a sys.path or pkg.__path__ item. It should return 
> >     an "importer" object (see below), or raise ImportError 
> >     or return None if it can't deal with the path item. By 
> >     default, sys.path_hooks only contains the zipimporter 
> >     type, if the zipimport module is available. 
> > 
> > Why have it raise ImportError or return None if it can't deal with the path
> > item?  Shouldn't there be only one way to report failure?  (Or do I misread
> > your explanation?)
> 
> No, you read correctly... I'm not sure, I just thought if might be
> convenient to also be able to return None. I'd be happy to stick
> with just one way, though (which would be raising ImportError, as
> that's easiest from __init__ methods).

Hm, I'd prefer None.  With an exception (especially when you're
reusing an existing exception) you never know if it was raised
intentionally or whether it means a real (unexpected) error -- in the
latter case, swallowing the traceback would be a bad idea because it
makes diagnosing the real problem hard.  ("Why does my zipimport not
work?  I don't get any errors, but it doesn't work...")

--Guido van Rossum (home page: http://www.python.org/~guido/)



From edcjones@erols.com  Thu Dec 12 16:43:42 2002
From: edcjones@erols.com (Edward C. Jones)
Date: Thu, 12 Dec 2002 11:43:42 -0500
Subject: [Python-Dev] os.path.walk generator
Message-ID: <3DF8BCBE.9000608@erols.com>

I feel that "The Right Way" to walk through a directory tree is with a 
generator. Therefore I suggest that a generator

walk2(topdir, hidden=None)

be added to os.path. It yields all the files and directories under (and 
including) "topdir". If "hidden" is true, then hidden files and 
directories are included.

Thanks,
Ed Jones



From just@letterror.com  Thu Dec 12 16:25:56 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 12 Dec 2002 17:25:56 +0100
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: <200212121556.gBCFu5105528@odiug.zope.com>
Message-ID: <r01050400-1022-6A8674AA0DEE11D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> Hm, I'd prefer None. 

The reason I prefer an exception (any exception, could be a new one) is that you
can't return None from an __init__ method, and a class/type is the *perfect*
candidate for a hook (zipimport.zipimporter is one, too). I'd prefer it if
people didn't have to write a __new__ method or a separate factory function.

> With an exception (especially when you're
> reusing an existing exception) you never know if it was raised
> intentionally or whether it means a real (unexpected) error -- in the
> latter case, swallowing the traceback would be a bad idea because it
> makes diagnosing the real problem hard.  ("Why does my zipimport not
> work?  I don't get any errors, but it doesn't work...")

(I think import.c should print a warning if the zipimport hook couldn't be
installed, similar to the site.py warning. The hook itself doesn't invoke any
Python code, so an ImportError is a 100% sure sign zipimporter can't handle the
path.)

Unless you do imports *in* the function/class __init__ that is the hook, there's
no chance of getting ImportErrors after the hook is installed, so I'm not too
worried.

The problem is perhaps comparable to old-style sequence iteration: the
__getitem__ implementation *could* raise a different IndexError than the author
intended.

Just


From aahz@pythoncraft.com  Thu Dec 12 16:26:34 2002
From: aahz@pythoncraft.com (Aahz)
Date: Thu, 12 Dec 2002 11:26:34 -0500
Subject: [Python-Dev] os.path.walk generator
In-Reply-To: <3DF8BCBE.9000608@erols.com>
References: <3DF8BCBE.9000608@erols.com>
Message-ID: <20021212162634.GA29004@panix.com>

On Thu, Dec 12, 2002, Edward C. Jones wrote:
>
> I feel that "The Right Way" to walk through a directory tree is with a 
> generator. Therefore I suggest that a generator
> 
> walk2(topdir, hidden=None)
> 
> be added to os.path. It yields all the files and directories under (and 
> including) "topdir". If "hidden" is true, then hidden files and 
> directories are included.

python-dev is the wrong place for this discussion; you should post to
comp.lang.python and/or submit a feature request to SourceForge.  Best
of all, post a patch to SF.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From mal@lemburg.com  Thu Dec 12 16:27:42 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 17:27:42 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>	<2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com>	<2mn0nbo48m.fsf@starship.python.net> <3DF87F5C.7060504@lemburg.com> <00ef01c2a1e1$e998bd50$741ee8d9@mira>
Message-ID: <3DF8B8FE.7090905@lemburg.com>

Martin v. L=F6wis wrote:
>>Ok, I've narrowed down the problem to the missing shared mods
>>struct and _codecs. It seems that the distutils setup.py is
>>not run before installing and compiling the .py files.
>>
>>There are two options:
>>1. tweak the Makefile targets
>>2. make struct and _codecs built-in modules
>>
>>I think the second option is more attractive.
>=20
>=20
> There is a third option:
>=20
> 3. Document that the proper build procedure is "configure;make;make
> install"

That's what I did. It still fails: the reason is that shared mods
are built *after* python itself is built and since struct and
_codecs are shared mods, compiling the .py files doesn't work
since that's done when building the python target (before building
the shared mods).

I'm +1 on option 2.

> Actually, that is documented, so I'd vote "pilot error".
>=20
> That said, I'm always in favour of making more modules builtin again, a=
s
> this simplifies the build process and may slightly improve performance
> if the module is imported.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From guido@python.org  Thu Dec 12 16:28:54 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 11:28:54 -0500
Subject: [Python-Dev] os.path.walk generator
In-Reply-To: Your message of "Thu, 12 Dec 2002 11:43:42 EST."
 <3DF8BCBE.9000608@erols.com>
References: <3DF8BCBE.9000608@erols.com>
Message-ID: <200212121628.gBCGSs005830@odiug.zope.com>

> I feel that "The Right Way" to walk through a directory tree is with a 
> generator. Therefore I suggest that a generator
> 
> walk2(topdir, hidden=None)
> 
> be added to os.path. It yields all the files and directories under (and 
> including) "topdir". If "hidden" is true, then hidden files and 
> directories are included.

Submit a patch.

I think hiding 'hidden' files should not be up to the walk() generator
-- os.listdir() and the existing os.path.walk() don't hide them
either.

Style nit: don't use None to indicate False.  Use False.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Paul.Moore@atosorigin.com  Thu Dec 12 16:32:14 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Thu, 12 Dec 2002 16:32:14 -0000
Subject: [Python-Dev] new import hooks & zip import
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861997F@UKDCX001.uk.int.atosorigin.com>

Guido wrote:

> > Skip Montanaro wrote:
> >=20
> > > Scanning the patch summary I see:
> > >=20
> > >     sys.path_hooks is a list of callable objects that take=20
> > >     a string as their only argument. A hook will be called=20
> > >     with a sys.path or pkg.__path__ item. It should return=20
> > >     an "importer" object (see below), or raise ImportError=20
> > >     or return None if it can't deal with the path item. By=20
> > >     default, sys.path_hooks only contains the zipimporter =20
> > >     type, if the zipimport module is available.           =20
> > >
> > > Why have it raise ImportError or return None if it can't
> > > deal with the path item? Shouldn't there be only one way to
> > > report failure? (Or do I misread your explanation?)
> >
> > No, you read correctly... I'm not sure, I just thought if
> > might be convenient to also be able to return None. I'd
> > be happy to stick with just one way, though (which would
> > be raising ImportError, as that's easiest from __init__
> > methods).
>
> Hm, I'd prefer None. With an exception (especially when you're
> reusing an existing exception) you never know if it was raised
> intentionally or whether it means a real (unexpected) error --
> in the latter case, swallowing the traceback would be a bad
> idea because it makes diagnosing the real problem hard. ("Why
> does my zipimport not work? I don't get any errors, but it
> doesn't work...")

The problem is that in 99% of cases (and specifically for
zipimporter) the "callable object" in question is a class. It's
not possible for a class when called to return a value (the
constructor either does its job or raises an exception). So an
exception signalling "I don't apply to this path item" is the
only way of letting classes be put directly on sys.path_hooks.
You could mandate wrappers, but I imagine that

    def my_importer_wrapper(path):
        try:
            return my_importer(path)
        except ImportError:
            return None

    sys.path_hooks.append(my_importer_wrapper)

is even more offensive...

Actually, I'd say that "I can't handle this" should *always* be
signalled by an exception (Only One Way To Do It), but rather than
reusing ImportError, that should be a new exception type (I don't
know what to call it though - ImportHandlerError?) That way, you
don't risk masking real exceptions.

Paul.


From mal@lemburg.com  Thu Dec 12 17:32:41 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 18:32:41 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>	<2mpts7o5r4.fsf@starship.python.net> <3DF86F21.6070509@lemburg.com>	<2mn0nbo48m.fsf@starship.python.net> <3DF87F5C.7060504@lemburg.com>	<00ef01c2a1e1$e998bd50$741ee8d9@mira> <3DF8B8FE.7090905@lemburg.com>
Message-ID: <3DF8C839.4040603@lemburg.com>

M.-A. Lemburg wrote:
> Martin v. L=F6wis wrote:
>=20
>>> Ok, I've narrowed down the problem to the missing shared mods
>>> struct and _codecs. It seems that the distutils setup.py is
>>> not run before installing and compiling the .py files.
>>>
>>> There are two options:
>>> 1. tweak the Makefile targets
>>> 2. make struct and _codecs built-in modules
>>>
>>> I think the second option is more attractive.
>>
>> There is a third option:
>>
>> 3. Document that the proper build procedure is "configure;make;make
>> install"
>=20
> That's what I did. It still fails:=20

Ah, didn't see the separate "make;" in there. You're right, Martin,
I ran "./configure; make install" using a script. The analysis
below applies to this way of invoking the make process.

> the reason is that shared mods
> are built *after* python itself is built and since struct and
> _codecs are shared mods, compiling the .py files doesn't work
> since that's done when building the python target (before building
> the shared mods).

I'll check in a fix which makes _codecs a builtin and removes the
struct dependency from codecs.py.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Thu Dec 12 17:41:04 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 18:41:04 +0100
Subject: [Python-Dev] os.path.walk generator
In-Reply-To: <20021212162634.GA29004@panix.com>
References: <3DF8BCBE.9000608@erols.com> <20021212162634.GA29004@panix.com>
Message-ID: <m34r9j2kzz.fsf@mira.informatik.hu-berlin.de>

Aahz <aahz@pythoncraft.com> writes:

> > walk2(topdir, hidden=None)
> > 
> > be added to os.path. It yields all the files and directories under (and 
> > including) "topdir". If "hidden" is true, then hidden files and 
> > directories are included.
> 
> python-dev is the wrong place for this discussion; you should post to
> comp.lang.python and/or submit a feature request to SourceForge.  Best
> of all, post a patch to SF.

I'd take this back a bit. python-dev is the wrong place to ask for new
features, yes. However, it is ok to ask "Might it be acceptable if I
implement it?", to which possible answers are "yes", "no", or "write a
PEP".

For that matter, I'd like to steer attention to
python.org/sf/619222. I think any walk modification should take file
types into account (a feature which isn't implemented, either). I hope
that Python adds support for file types one day, and it would be
unfortunate if that would cause introduction of walk3 and walk4.

Please take into consideration that the os.path.walk callback can
control traversal, by modifying the file list. It should be specified
if this is also supported for walk2.

Regards,
Martin


From mal@lemburg.com  Thu Dec 12 17:43:01 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 12 Dec 2002 18:43:01 +0100
Subject: [Python-Dev] make install fails with current Python CVS
References: <3DF86660.6030808@lemburg.com>	<00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>	<3DF86EB6.1020200@lemburg.com> <00e301c2a1e1$25fa9d50$741ee8d9@mira>
Message-ID: <3DF8CAA5.1040905@lemburg.com>

Martin v. L=F6wis wrote:
>>>>>Sorry: MemoryError:
>>
>>Could this error message be made more helpful :-)
>=20
> I don't know. I tried, and failed. I did not try too hard, though.
>=20
> I just knew what the problem was as I investigated this a couple of
> times before, to track it down to the codecs lookup facility not
> working.

Could you tell me where this error is generated ?

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Thu Dec 12 17:54:32 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 18:54:32 +0100
Subject: [Python-Dev] make install fails with current Python CVS
In-Reply-To: <3DF8CAA5.1040905@lemburg.com>
References: <3DF86660.6030808@lemburg.com>
 <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com>
 <3DF86EB6.1020200@lemburg.com> <00e301c2a1e1$25fa9d50$741ee8d9@mira>
 <3DF8CAA5.1040905@lemburg.com>
Message-ID: <m3r8cn15t3.fsf@mira.informatik.hu-berlin.de>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> > I don't know. I tried, and failed. I did not try too hard, though.
> > I just knew what the problem was as I investigated this a couple of
> > times before, to track it down to the codecs lookup facility not
> > working.
> 
> Could you tell me where this error is generated ?

That's part of the problem: It was difficult to find, and I forgot
now. It may be one of the parser error codes.

Regards,
Martin


From mwh@python.net  Thu Dec 12 17:58:29 2002
From: mwh@python.net (Michael Hudson)
Date: 12 Dec 2002 17:58:29 +0000
Subject: [Python-Dev] make install fails with current Python CVS
In-Reply-To: "M.-A. Lemburg"'s message of "Thu, 12 Dec 2002 18:43:01 +0100"
References: <3DF86660.6030808@lemburg.com> <00bb01c2a1cb$6bd13160$741ee8d9@mira> <3DF86C86.7000508@lemburg.com> <3DF86EB6.1020200@lemburg.com> <00e301c2a1e1$25fa9d50$741ee8d9@mira> <3DF8CAA5.1040905@lemburg.com>
Message-ID: <2mk7ifnmpm.fsf@starship.python.net>

"M.-A. Lemburg" <mal@lemburg.com> writes:

> Martin v. L=F6wis wrote:
> >>>>>Sorry: MemoryError:
> >>
> >>Could this error message be made more helpful :-)
> >=20
> > I don't know. I tried, and failed. I did not try too hard, though.
> >=20
> > I just knew what the problem was as I investigated this a couple of
> > times before, to track it down to the codecs lookup facility not
> > working.
> 
> Could you tell me where this error is generated ?

I imagine it's somewhere inside PyUnicode_AsEncodedString which is
called from compile.c:parsestr.  If you want to expose the latter to
Python while you're at it, be my guest (sf feature request 617979).

Cheers,
M.

-- 
  I've even been known to get Marmite *near* my mouth -- but never
  actually in it yet.  Vegamite is right out.
 UnicodeError: ASCII unpalatable error: vegamite found, ham expected
                                       -- Tim Peters, comp.lang.python


From dave@boost-consulting.com  Thu Dec 12 18:01:57 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Thu, 12 Dec 2002 13:01:57 -0500
Subject: [Python-Dev] Possible low-hanging optimization
Message-ID: <ud6o7une2.fsf@boost-consulting.com>

In Boost.Python, extension class instance dictionaries are created
only on-demand.  It's an obvious win for lots of wrapped C++ class
instances, which will never have Python attributes tacked on.  This
also seems like a potential big win for some kinds of Python
instances, like those which use __slots__ but may also have a
dictionary (IIRC that is a possible configuration).  The code to do
this in Boost.Python is small and "just works" transparently, so I
thought there might be some interest here for the Python core:

I've "C-ified" the code here, sort of on-the-fly, just to give an
idea, but it's so simple that any errors can't obscure the intent
/too/ much ;-)

      static PyObject* instance_get_dict(PyObject* op, void*)
      {
          bpl_instance* inst = (bpl_instance*)op;
          if (inst->dict == 0)
              inst->dict = PyDict_New();
          Py_XINCREF(inst->dict);
          return inst->dict;
      }
    
      static int instance_set_dict(PyObject* op, PyObject* dict, void*)
      {
          (bpl_instance)* inst = (bpl_instance*)op;
          Py_XDECREF(inst->dict);
          Py_XINCREF(dict);
          inst->dict = dict
          return 0;
      }

  static PyGetSetDef instance_getsets[] = { /* used as tp_getset */
      {"__dict__",  instance_get_dict,  instance_set_dict, NULL},
      {0}
  };

Useful?

-Dave

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From gward@python.net  Thu Dec 12 17:34:10 2002
From: gward@python.net (Greg Ward)
Date: Thu, 12 Dec 2002 12:34:10 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib textwrap.py,1.18,1.19
In-Reply-To: <m3y96wcx97.fsf@mira.informatik.hu-berlin.de>
References: <E18LQgZ-00070M-00@sc8-pr-cvs1.sourceforge.net> <r01050400-1022-A7D025780CEC11D7AD06003065D5E7E4@[10.0.0.23]> <20021211134958.GA10384@cthulhu.gerg.ca> <017b01c2a11e$4b655c60$0900a8c0@spiff> <20021211143957.GA10597@cthulhu.gerg.ca> <m3y96wcx97.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021212173410.GA1549@cthulhu.gerg.ca>

On 11 December 2002, Martin v. Löwis said:
> Please, don't assume any specific encoding. Why is Latin-1 better than
> KOI8-R? The only encoding that is truly better than all others is
> ASCII, since virtually all other encodings have ASCII as a subset
> (except for the EBCDIC ones, and, with limitations, the ISO-2022
> ones).

Sorry, I should have said *Western* European languages.

> [If you meant to suggest no specific processing for &nbsp; disregard
> this comment]

Correct: 0xa0 (the ISO-8859-1 non-breaking space) is now just another
character, no matter what your locale says.

OK, here's a proposed policy statement for textwrap's I18N support (or
lack thereof):

  textwrap is intended for wrapping ASCII-encoded English-language text
  for use in fixed-width settings such as email messages and plain text
  files.  It also works for Unicode strings that only contain ASCII
  characters.  And it should work for other European languages that use
  Latin script and have similar line-breaking conventions for monospaced
  text as English.  textwrap is not intended for use with languages or
  character sets that do not use Latin characters.

I should add that to the docs.  (Should probably also mention that it
doesn't do hyphenation -- speaking of huge cans of worms!)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
This quote intentionally left blank.


From jacobs@penguin.theopalgroup.com  Thu Dec 12 18:06:17 2002
From: jacobs@penguin.theopalgroup.com (Kevin Jacobs)
Date: Thu, 12 Dec 2002 13:06:17 -0500 (EST)
Subject: [Python-Dev] Possible low-hanging optimization
In-Reply-To: <ud6o7une2.fsf@boost-consulting.com>
Message-ID: <Pine.LNX.4.44.0212121304020.6584-100000@penguin.theopalgroup.com>

On Thu, 12 Dec 2002, David Abrahams wrote:
> In Boost.Python, extension class instance dictionaries are created
> only on-demand.  It's an obvious win for lots of wrapped C++ class
> instances, which will never have Python attributes tacked on.  This
> also seems like a potential big win for some kinds of Python
> instances, like those which use __slots__ but may also have a
> dictionary (IIRC that is a possible configuration).

I am fairly confident that CPython doesn't allocate instance dictionaries
until they are needed.  At one point I verified this, though my brain and
the code has likely diverged much since there.

-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com



From guido@python.org  Thu Dec 12 18:07:45 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 13:07:45 -0500
Subject: [Python-Dev] Possible low-hanging optimization
In-Reply-To: Your message of "Thu, 12 Dec 2002 13:01:57 EST."
 <ud6o7une2.fsf@boost-consulting.com>
References: <ud6o7une2.fsf@boost-consulting.com>
Message-ID: <200212121807.gBCI7jL20642@odiug.zope.com>

> In Boost.Python, extension class instance dictionaries are created
> only on-demand.  It's an obvious win for lots of wrapped C++ class
> instances, which will never have Python attributes tacked on.  This
> also seems like a potential big win for some kinds of Python
> instances, like those which use __slots__ but may also have a
> dictionary (IIRC that is a possible configuration).  The code to do
> this in Boost.Python is small and "just works" transparently, so I
> thought there might be some interest here for the Python core:
> 
> I've "C-ified" the code here, sort of on-the-fly, just to give an
> idea, but it's so simple that any errors can't obscure the intent
> /too/ much ;-)
> 
>       static PyObject* instance_get_dict(PyObject* op, void*)
>       {
>           bpl_instance* inst = (bpl_instance*)op;
>           if (inst->dict == 0)
>               inst->dict = PyDict_New();
>           Py_XINCREF(inst->dict);
>           return inst->dict;
>       }
>     
>       static int instance_set_dict(PyObject* op, PyObject* dict, void*)
>       {
>           (bpl_instance)* inst = (bpl_instance*)op;
>           Py_XDECREF(inst->dict);
>           Py_XINCREF(dict);
>           inst->dict = dict
>           return 0;
>       }
> 
>   static PyGetSetDef instance_getsets[] = { /* used as tp_getset */
>       {"__dict__",  instance_get_dict,  instance_set_dict, NULL},
>       {0}
>   };
> 
> Useful?

AFAICT, this is already done for new-style classes.  For classic
classes, it would be too much of a break with history.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From dave@boost-consulting.com  Thu Dec 12 18:11:17 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Thu, 12 Dec 2002 13:11:17 -0500
Subject: [Python-Dev] Possible low-hanging optimization
In-Reply-To: <200212121807.gBCI7jL20642@odiug.zope.com> (Guido van Rossum's
 message of "Thu, 12 Dec 2002 13:07:45 -0500")
References: <ud6o7une2.fsf@boost-consulting.com>
 <200212121807.gBCI7jL20642@odiug.zope.com>
Message-ID: <uy96vt8e2.fsf@boost-consulting.com>

Guido van Rossum <guido@python.org> writes:

> AFAICT, this is already done for new-style classes.  For classic
> classes, it would be too much of a break with history.

Hmm, I wonder if my extension classes will inherit this from the
new-style classes automatically.  Was this feature in 2.2 as released?

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From martin@v.loewis.de  Thu Dec 12 18:18:37 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 19:18:37 +0100
Subject: [Python-Dev] Tcl, Tkinter, and threads
Message-ID: <m3isxz14oy.fsf@mira.informatik.hu-berlin.de>

I have now applied changes to _tkinter to make it work when Tcl was
built with --enable-threads. For those of you interested in such kind
of stuff, here is the full story.

This work was triggered by the bug report

http://bugs.debian.org/171353

where pydoc -g crashes Tcl. It turns out that pydoc uses threads, and
invokes a button configure command from a thread different from the
one where the Tcl interpreter was called. Tcl uses thread-local
storage to associate a TkDisplay* with a X Display*, and that TLS was
uninitialized in this other thread.

Discussion with Tcl people at

http://sourceforge.net/tracker/index.php?func=detail&aid=649209&group_id=12997&atid=112997

lead to the conclusion that this is intentional; they call it the
"appartment model"; you can use the Tcl interpreter only from the
thread that has created it, you cannot pass Tcl_Obj* across thread
boundaries, and you cannot invoke Tcl commands from other threads.

If Tcl is not built for threads, this all is not relevant, since there
is only a single set of "thread-local" data (i.e. it isn't thread-local);
the thread API is still there, anyway.

To overcome this limitation, I now use Tcl Queues to pass commands
from one thread the other; this is also how the Tcl thread extension
works. You allocate a Tcl_Event, fill some data, put it into a queue,
alert the target thread, and block on a Tcl_Condition. The target
thread fetches the event from the queue, invokes the callback
function, which performs the Tcl action, and notifies the condition.

Passing of results happens to stack variables in the calling thread
whose addresses are put into the event.

This kind of marshalling now happens for the following APIs:
- call: passes the PyObject* args to the target thread. There it
  is converted into Tcl objects, the command is invoked, and the result
  is converted back to a Python object.
- getvar/setvar/unsetvar: pass the variable name and value, and invoke
  the Tcl API in the target thread.
- createcommand/deletecommand: likewise.

For a few APIs, this marshalling is not possible in principle:
- mainloop/doonevent: event processing must happen in the interpreter
  thread, by nature of the Tcl threading model

For a larger number of APIs, I found the effort not worthwhile:
- globalcall, eval, globaleval, evalfile, record, adderrorinfo,
  exprstring, exprlong, exprdouble, exprboolean, createfilehandler,
  deletefilehandler

For all these functions, _tkinter will now raise an exception if they
are invoked in the wrong thread.

A tricky question is what happens if the target thread is not
processing events right now, either because it mainloop hasn't been
invoked, or because it is busy doing something else. The current code
raises an exception if the target interpreter is not in the mainloop,
and blocks (potentially indefinitely) if the target process does not
unqueue its events.

This might cause problems if you create multiple interpreters in one
thread: it would be sufficient if one of them processes
events. Currently, calls to the other interpreters will raise the
exception that the mainloop has not been entered. I hope this won't
cause problems in practice, since you rarely have more than one
interpreter.

With these changes, it would now be possible to build Tcl in threaded
mode on Windows. This has both advantages and disadvantages:
+ It allows to get rid of the Tcl lock, and the nearly-busy wait.
- It may cause problems for existing applications if they run into
  one of the limitations. Of course, those applications will run into
  the same limitations on Unix if Tcl was build with threads enabled.

If Tcl wasn't build with threads enabled, behaviour is nearly
unmodified.  Python will still invoke the Tcl thread API, but that
won't do anything, so there should be no user-visible change.

Regards,
Martin


From guido@python.org  Thu Dec 12 18:18:56 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 13:18:56 -0500
Subject: [Python-Dev] Possible low-hanging optimization
In-Reply-To: Your message of "Thu, 12 Dec 2002 13:11:17 EST."
 <uy96vt8e2.fsf@boost-consulting.com>
References: <ud6o7une2.fsf@boost-consulting.com> <200212121807.gBCI7jL20642@odiug.zope.com>
 <uy96vt8e2.fsf@boost-consulting.com>
Message-ID: <200212121818.gBCIIuE20736@odiug.zope.com>

> > AFAICT, this is already done for new-style classes.  For classic
> > classes, it would be too much of a break with history.
> 
> Hmm, I wonder if my extension classes will inherit this from the
> new-style classes automatically.  Was this feature in 2.2 as released?

I think so, but you better check to be sure.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From theller@python.net  Thu Dec 12 18:34:52 2002
From: theller@python.net (Thomas Heller)
Date: 12 Dec 2002 19:34:52 +0100
Subject: [Python-Dev] PEP 298
Message-ID: <y96vaxwz.fsf@python.net>

I intend to work on the PEP 298 (locked buffer interface)
implementation, with the hope that it is not too late to bring this
into 2.3.  Is there still a chance?

If it will not make it into 2.3, I will retract the PEP,
because 2.4 is too late for me.

Thomas



From guido@python.org  Thu Dec 12 18:39:50 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 13:39:50 -0500
Subject: [Python-Dev] PEP 298
In-Reply-To: Your message of "12 Dec 2002 19:34:52 +0100."
 <y96vaxwz.fsf@python.net>
References: <y96vaxwz.fsf@python.net>
Message-ID: <200212121839.gBCIdoM21455@odiug.zope.com>

> I intend to work on the PEP 298 (locked buffer interface)
> implementation, with the hope that it is not too late to bring this
> into 2.3.  Is there still a chance?
> 
> If it will not make it into 2.3, I will retract the PEP,
> because 2.4 is too late for me.

Is the PEP accepted?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From theller@python.net  Thu Dec 12 18:44:23 2002
From: theller@python.net (Thomas Heller)
Date: 12 Dec 2002 19:44:23 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <200212121839.gBCIdoM21455@odiug.zope.com>
References: <y96vaxwz.fsf@python.net>
 <200212121839.gBCIdoM21455@odiug.zope.com>
Message-ID: <smx3axh4.fsf@python.net>

Guido van Rossum <guido@python.org> writes:

> > I intend to work on the PEP 298 (locked buffer interface)
> > implementation, with the hope that it is not too late to bring this
> > into 2.3.  Is there still a chance?
> > 
> > If it will not make it into 2.3, I will retract the PEP,
> > because 2.4 is too late for me.
> 
> Is the PEP accepted?

How could I know? Marked as Draft, I would say it is ready for pronouncement.
Only the implementation is missing.

Thomas



From guido@python.org  Thu Dec 12 18:51:46 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 13:51:46 -0500
Subject: [Python-Dev] PEP 298
In-Reply-To: Your message of "12 Dec 2002 19:44:23 +0100."
 <smx3axh4.fsf@python.net>
References: <y96vaxwz.fsf@python.net> <200212121839.gBCIdoM21455@odiug.zope.com>
 <smx3axh4.fsf@python.net>
Message-ID: <200212121851.gBCIpkh21556@odiug.zope.com>

> Guido van Rossum <guido@python.org> writes:
> 
> > > I intend to work on the PEP 298 (locked buffer interface)
> > > implementation, with the hope that it is not too late to bring this
> > > into 2.3.  Is there still a chance?
> > > 
> > > If it will not make it into 2.3, I will retract the PEP,
> > > because 2.4 is too late for me.
> > 
> > Is the PEP accepted?
> 
> How could I know? Marked as Draft, I would say it is ready for
> pronouncement.  Only the implementation is missing.

There are doubts about its usefulness.  (AKA YAGNI.)  Would you also
provide implementations for common object types?  Otherwise there
wouldn't be a point, right?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Thu Dec 12 18:56:23 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 13:56:23 -0500
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: Your message of "Thu, 12 Dec 2002 16:32:14 GMT."
 <16E1010E4581B049ABC51D4975CEDB8861997F@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB8861997F@UKDCX001.uk.int.atosorigin.com>
Message-ID: <200212121856.gBCIuNG21865@odiug.zope.com>

> Guido wrote:
> 
> > > Skip Montanaro wrote:
> > > 
> > > > Scanning the patch summary I see:
> > > > 
> > > >     sys.path_hooks is a list of callable objects that take 
> > > >     a string as their only argument. A hook will be called 
> > > >     with a sys.path or pkg.__path__ item. It should return 
> > > >     an "importer" object (see below), or raise ImportError 
> > > >     or return None if it can't deal with the path item. By 
> > > >     default, sys.path_hooks only contains the zipimporter  
> > > >     type, if the zipimport module is available.            
> > > >
> > > > Why have it raise ImportError or return None if it can't
> > > > deal with the path item? Shouldn't there be only one way to
> > > > report failure? (Or do I misread your explanation?)
> > >
> > > No, you read correctly... I'm not sure, I just thought if
> > > might be convenient to also be able to return None. I'd
> > > be happy to stick with just one way, though (which would
> > > be raising ImportError, as that's easiest from __init__
> > > methods).
> >
> > Hm, I'd prefer None. With an exception (especially when you're
> > reusing an existing exception) you never know if it was raised
> > intentionally or whether it means a real (unexpected) error --
> > in the latter case, swallowing the traceback would be a bad
> > idea because it makes diagnosing the real problem hard. ("Why
> > does my zipimport not work? I don't get any errors, but it
> > doesn't work...")

[Paul Moore]
> The problem is that in 99% of cases (and specifically for
> zipimporter) the "callable object" in question is a class. It's
> not possible for a class when called to return a value (the
> constructor either does its job or raises an exception). So an
> exception signalling "I don't apply to this path item" is the
> only way of letting classes be put directly on sys.path_hooks.
> You could mandate wrappers, but I imagine that
> 
>     def my_importer_wrapper(path):
>         try:
>             return my_importer(path)
>         except ImportError:
>             return None
> 
>     sys.path_hooks.append(my_importer_wrapper)
> 
> is even more offensive...
> 
> Actually, I'd say that "I can't handle this" should *always* be
> signalled by an exception (Only One Way To Do It), but rather than
> reusing ImportError, that should be a new exception type (I don't
> know what to call it though - ImportHandlerError?) That way, you
> don't risk masking real exceptions.

OK, point taken.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From theller@python.net  Thu Dec 12 19:02:24 2002
From: theller@python.net (Thomas Heller)
Date: 12 Dec 2002 20:02:24 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <200212121851.gBCIpkh21556@odiug.zope.com>
References: <y96vaxwz.fsf@python.net>
 <200212121839.gBCIdoM21455@odiug.zope.com> <smx3axh4.fsf@python.net>
 <200212121851.gBCIpkh21556@odiug.zope.com>
Message-ID: <n0nbawn3.fsf@python.net>

Guido van Rossum <guido@python.org> writes:

> There are doubts about its usefulness.  (AKA YAGNI.)  Would you also
> provide implementations for common object types?  Otherwise there
> wouldn't be a point, right?

Not really, IMO. It is an interface, not an implementation (except that
there are 3 helper functions in abstract.c).

I have patches ready for string and unicode objects (trivial), and
started a patch for arrayobject. This is more involved, because the
array must be locked while a buffer pointer is out somewhere.

I could also implement it for mmap objects, there are comparable issues.

Maybe the most useful object type would be the upcoming bytes object
(PEP 296).  It seems, this will not be included in 2.3, but it could also
be implemented as a separate extension - impossible for the locked buffer
interface, which must be included into the core.

If you call YAGNI, it's ok for me, that's why I'm asking. Better save
the time in this case...

Thomas



From guido@python.org  Thu Dec 12 19:10:50 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 14:10:50 -0500
Subject: [Python-Dev] PEP 298
In-Reply-To: Your message of "12 Dec 2002 20:02:24 +0100."
 <n0nbawn3.fsf@python.net>
References: <y96vaxwz.fsf@python.net> <200212121839.gBCIdoM21455@odiug.zope.com> <smx3axh4.fsf@python.net> <200212121851.gBCIpkh21556@odiug.zope.com>
 <n0nbawn3.fsf@python.net>
Message-ID: <200212121910.gBCJAo222634@odiug.zope.com>

> Guido van Rossum <guido@python.org> writes:
> 
> > There are doubts about its usefulness.  (AKA YAGNI.)  Would you also
> > provide implementations for common object types?  Otherwise there
> > wouldn't be a point, right?
> 
> Not really, IMO. It is an interface, not an implementation (except that
> there are 3 helper functions in abstract.c).
> 
> I have patches ready for string and unicode objects (trivial), and
> started a patch for arrayobject. This is more involved, because the
> array must be locked while a buffer pointer is out somewhere.
> 
> I could also implement it for mmap objects, there are comparable issues.
> 
> Maybe the most useful object type would be the upcoming bytes object
> (PEP 296).  It seems, this will not be included in 2.3, but it could also
> be implemented as a separate extension - impossible for the locked buffer
> interface, which must be included into the core.
> 
> If you call YAGNI, it's ok for me, that's why I'm asking. Better save
> the time in this case...

Well, I have no personal need for it.  If you're the only one, that
sounds like a big YAGNI to me...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Thu Dec 12 19:25:13 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 12 Dec 2002 20:25:13 +0100
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: <200212121856.gBCIuNG21865@odiug.zope.com>
Message-ID: <r01050400-1022-88A8BBA80E0711D7AD06003065D5E7E4@[10.0.0.23]>

[Paul Moore]
> > Actually, I'd say that "I can't handle this" should *always* be
> > signalled by an exception (Only One Way To Do It), but rather than
> > reusing ImportError, that should be a new exception type (I don't
> > know what to call it though - ImportHandlerError?) That way, you
> > don't risk masking real exceptions.

[GvR]
> OK, point taken.

If we should add a new exception, would it be ok for it to live in the imp
module? It would be a fairly specialized exception so it's kindof a shame to
pollute the builtin namespace for it.

ImportHookError?

zipimporter currently raises ZipImportError in this case, but that's a subclass
of ImportError... (I'm actually -0 on adding a new exception; I don't think
reusing ImportError is all that bad.)

Just


From skip@pobox.com  Thu Dec 12 19:47:53 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 12 Dec 2002 13:47:53 -0600
Subject: [Python-Dev] Possible low-hanging optimization
In-Reply-To: <200212121807.gBCI7jL20642@odiug.zope.com>
References: <ud6o7une2.fsf@boost-consulting.com>
 <200212121807.gBCI7jL20642@odiug.zope.com>
Message-ID: <15864.59369.517000.126606@montanaro.dyndns.org>

    Guido> For classic classes, it would be too much of a break with
    Guido> history.

In addition, since most classes exist to associate state with methods that
operate on it you'd probably only be delaying the creation of self.__dict__
by a few milliseconds in the common case.

Skip



From guido@python.org  Thu Dec 12 19:47:32 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 14:47:32 -0500
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: Your message of "Thu, 12 Dec 2002 20:25:13 +0100."
 <r01050400-1022-88A8BBA80E0711D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-88A8BBA80E0711D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212121947.gBCJlWf23496@odiug.zope.com>

> [Paul Moore]
> > > Actually, I'd say that "I can't handle this" should *always* be
> > > signalled by an exception (Only One Way To Do It), but rather than
> > > reusing ImportError, that should be a new exception type (I don't
> > > know what to call it though - ImportHandlerError?) That way, you
> > > don't risk masking real exceptions.
> 
> [GvR]
> > OK, point taken.
> 
> If we should add a new exception, would it be ok for it to live in the imp
> module? It would be a fairly specialized exception so it's kindof a shame to
> pollute the builtin namespace for it.
> 
> ImportHookError?
> 
> zipimporter currently raises ZipImportError in this case, but that's a subclass
> of ImportError... (I'm actually -0 on adding a new exception; I don't think
> reusing ImportError is all that bad.)

Let's try raising ImportError (or a subclass) for now.  It's okay for
it to live in the imp module.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From martin@v.loewis.de  Thu Dec 12 19:54:14 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 20:54:14 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <y96vaxwz.fsf@python.net>
References: <y96vaxwz.fsf@python.net>
Message-ID: <m31y4n109l.fsf@mira.informatik.hu-berlin.de>

Thomas Heller <theller@python.net> writes:

> I intend to work on the PEP 298 (locked buffer interface)
> implementation, with the hope that it is not too late to bring this
> into 2.3.  Is there still a chance?

Having looked at the PEP for the first time, I have a number of
comments:

- I think there must be a way to obtain the address of the buffer on
  the Python level, see

  http://tinyurl.com/3h79

  Using /dev/poll on Solaris requires a way to create a struct whose
  address you know.

- There must be a Python-level API to obtain a locked buffer.

- How will string objects implement that interface? In particular,
  what prevents the refcount of the string dropping to zero?

Regards,
Martin


From theller@python.net  Thu Dec 12 20:06:47 2002
From: theller@python.net (Thomas Heller)
Date: 12 Dec 2002 21:06:47 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
References: <y96vaxwz.fsf@python.net>
 <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
Message-ID: <bs3ratns.fsf@python.net>

> > I intend to work on the PEP 298 (locked buffer interface)
> > implementation, with the hope that it is not too late to bring this
> > into 2.3.  Is there still a chance?
> 
> Having looked at the PEP for the first time, I have a number of
> comments:
[Martin, I rearranged your questions]

> - How will string objects implement that interface? In particular,
>   what prevents the refcount of the string dropping to zero?
> 

It is required to unlock the buffer again after you don't need it
any longer. You do this by calling the releaselockedbuffer() function
with the pointer to the original python object. So the object haas to
keep alive.

> 
> - I think there must be a way to obtain the address of the buffer on
>   the Python level, see
> 
>   http://tinyurl.com/3h79
> 
>   Using /dev/poll on Solaris requires a way to create a struct whose
>   address you know.

Hm, the usual hacky way is to use an array object and call buffer_info()
on it.

> 
> - There must be a Python-level API to obtain a locked buffer.
> 

That may be, but the PEP only describes the interface.  Extensions
may implement a 'locked buffer object', or whatever, and expose the
address to Python. Dangerous, though, because then you can separate
the python oobject pointer and the buffer pointer.

Thomas



From martin@v.loewis.de  Thu Dec 12 20:18:52 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 21:18:52 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <bs3ratns.fsf@python.net>
References: <y96vaxwz.fsf@python.net>
 <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
 <bs3ratns.fsf@python.net>
Message-ID: <m3of7ryor7.fsf@mira.informatik.hu-berlin.de>

Thomas Heller <theller@python.net> writes:

> > - How will string objects implement that interface? In particular,
> >   what prevents the refcount of the string dropping to zero?
> > 
> 
> It is required to unlock the buffer again after you don't need it
> any longer. You do this by calling the releaselockedbuffer() function
> with the pointer to the original python object. So the object haas to
> keep alive.

This is flawed. According to the PEP, an object which has
acquirelockedbuffer called will live forever:

# Retrieving a buffer from an object puts this object in a locked
# state during which the buffer may not be freed, resized, or
# reallocated.

But now you are saying that the object can be released even in locked
state.

> >   Using /dev/poll on Solaris requires a way to create a struct whose
> >   address you know.
> 
> Hm, the usual hacky way is to use an array object and call buffer_info()
> on it.

I see. I didn't know that. However, having to copy a string to an
array to obtain its address is not obvious; putting it into the buffer
objects seems more logical - especially if the buffer claims to be
*locked*.

> That may be, but the PEP only describes the interface.  Extensions
> may implement a 'locked buffer object', or whatever, and expose the
> address to Python. Dangerous, though, because then you can separate
> the python oobject pointer and the buffer pointer.

If it is just the interface, I'm -1. You don't need a PEP to define an
interface in Python - just establish an interface in the types you
care about. If you really mean to implement it primarily for some type
of yours, just do so (it' can't be *exactly* this interface, but you
can have the same operations, just in a different place).

Regards,
Martin


From guido@python.org  Thu Dec 12 20:21:01 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 15:21:01 -0500
Subject: [Python-Dev] Tcl, Tkinter, and threads
In-Reply-To: Your message of "12 Dec 2002 19:18:37 +0100."
 <m3isxz14oy.fsf@mira.informatik.hu-berlin.de>
References: <m3isxz14oy.fsf@mira.informatik.hu-berlin.de>
Message-ID: <200212122021.gBCKL1T24010@odiug.zope.com>

> I have now applied changes to _tkinter to make it work when Tcl was
> built with --enable-threads. For those of you interested in such kind
> of stuff, here is the full story.

Good work.  Thanks for taking over maintenance of _tkinter!

--Guido van Rossum (home page: http://www.python.org/~guido/)


From theller@python.net  Thu Dec 12 20:22:10 2002
From: theller@python.net (Thomas Heller)
Date: 12 Dec 2002 21:22:10 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
References: <y96vaxwz.fsf@python.net>
 <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
Message-ID: <65tzasy5.fsf@python.net>

martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes:

> Having looked at the PEP for the first time, I have a number of
> comments:
> 
> - I think there must be a way to obtain the address of the buffer on
>   the Python level, see
> 
>   http://tinyurl.com/3h79
> 
>   Using /dev/poll on Solaris requires a way to create a struct whose
>   address you know.

Do you know my ctypes module? http://tinyurl.com/2tyx

It allows to create Python objects which internally look like C data
types. Able to access and create structures, unions, pointers, and
such. And you do it (IMO) in a natural way.

The released package currently only runs under windows, although I have
a prototype working under Linux.

Thomas



From guido@python.org  Thu Dec 12 20:26:07 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 15:26:07 -0500
Subject: [Python-Dev] Tarfile support
In-Reply-To: Your message of "Thu, 12 Dec 2002 11:24:40 -0200."
 <20021212112440.A25044@ibook.distro.conectiva>
References: <20021212112440.A25044@ibook.distro.conectiva>
Message-ID: <200212122026.gBCKQ7x24337@odiug.zope.com>

> Lars Gustabel has opened a patch (#651082) with the implementation of
> Tarfile, including documentation and everything. I have already reviewed
> and sent him some considerations which were fixed (besides the
> constructor stuff we discussed here). It would be nice if other people
> reviewed it as well.

No time to review the code, but the feature set and docs look very
good.

Is there any chance that this can be contributed to the PSF under the
standard PSF license?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From niemeyer@conectiva.com  Thu Dec 12 20:29:54 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Thu, 12 Dec 2002 18:29:54 -0200
Subject: [Python-Dev] Tarfile support
In-Reply-To: <200212122026.gBCKQ7x24337@odiug.zope.com>
References: <20021212112440.A25044@ibook.distro.conectiva> <200212122026.gBCKQ7x24337@odiug.zope.com>
Message-ID: <20021212182954.A30768@ibook.distro.conectiva>

> No time to review the code, but the feature set and docs look very
> good.
> 
> Is there any chance that this can be contributed to the PSF under the
> standard PSF license?

Lars, can you please give us your position about this?

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From theller@python.net  Thu Dec 12 20:31:21 2002
From: theller@python.net (Thomas Heller)
Date: 12 Dec 2002 21:31:21 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <m3of7ryor7.fsf@mira.informatik.hu-berlin.de>
References: <y96vaxwz.fsf@python.net>
 <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
 <bs3ratns.fsf@python.net>
 <m3of7ryor7.fsf@mira.informatik.hu-berlin.de>
Message-ID: <znrb9dye.fsf@python.net>

martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=) writes:

> Thomas Heller <theller@python.net> writes:
> 
> > > - How will string objects implement that interface? In particular,
> > >   what prevents the refcount of the string dropping to zero?
> > > 
> > 
> > It is required to unlock the buffer again after you don't need it
> > any longer. You do this by calling the releaselockedbuffer() function
> > with the pointer to the original python object. So the object haas to
> > keep alive.
> 
> This is flawed. According to the PEP, an object which has
> acquirelockedbuffer called will live forever:
> 
> # Retrieving a buffer from an object puts this object in a locked
> # state during which the buffer may not be freed, resized, or
> # reallocated.
> 
> But now you are saying that the object can be released even in locked
> state.
> 

But the PEP it continues to say:

    The object must be unlocked again by releasing the buffer if it's
    no longer used by calling another function in the locked buffer
    interface.

Also, an easy solution to keep the string object alive forever
would be to Py_INCREF() in the acquirebuffer() function, and
Py_DECREF() in releasebuffer() again.

> > >   Using /dev/poll on Solaris requires a way to create a struct whose
> > >   address you know.
> > 
> > Hm, the usual hacky way is to use an array object and call buffer_info()
> > on it.
> 
> I see. I didn't know that. However, having to copy a string to an
> array to obtain its address is not obvious; putting it into the buffer
> objects seems more logical - especially if the buffer claims to be
> *locked*.
> 
> > That may be, but the PEP only describes the interface.  Extensions
> > may implement a 'locked buffer object', or whatever, and expose the
> > address to Python. Dangerous, though, because then you can separate
> > the python oobject pointer and the buffer pointer.
> 
> If it is just the interface, I'm -1. You don't need a PEP to define an
> interface in Python - just establish an interface in the types you
> care about. If you really mean to implement it primarily for some type
> of yours, just do so (it' can't be *exactly* this interface, but you
> can have the same operations, just in a different place).

That's what I have done, since I need it *now*, and in 2.2.

I just did not want the PEP to collect too much dust: it should
either be implemented or just die.

Once again: the whole point of the PEP is to avoid the flaws of the
'old' buffer interface.

Thomas



From martin@v.loewis.de  Thu Dec 12 20:42:17 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 21:42:17 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <znrb9dye.fsf@python.net>
References: <y96vaxwz.fsf@python.net>
 <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
 <bs3ratns.fsf@python.net>
 <m3of7ryor7.fsf@mira.informatik.hu-berlin.de>
 <znrb9dye.fsf@python.net>
Message-ID: <m3k7ifyno6.fsf@mira.informatik.hu-berlin.de>

Thomas Heller <theller@python.net> writes:

> But the PEP it continues to say:
> 
>     The object must be unlocked again by releasing the buffer if it's
>     no longer used by calling another function in the locked buffer
>     interface.

This is counter-intuitive, and under-specified. What does it mean that
"it's no longer used"? My code uses the buffer until system exit, so I
never need to release the buffer. I don't release the string, either,
since that was an element of ParseTuple, of which I only acquired the
buffer.

It doesn't talk about the object "being used", just "the buffer". The
spec should not prescribe under what conditions some API must be
invoked, but instead it should describe what happens if the API is
invoked.

> Also, an easy solution to keep the string object alive forever
> would be to Py_INCREF() in the acquirebuffer() function, and
> Py_DECREF() in releasebuffer() again.

I would prefer such an implementation strategy, and I would prefer if
the spec clarifies that the caller of acquire will keep the object
alive atleast until the corresponding release is called.

That, of course, means that it is no longer an option to not implement
release.

> > If it is just the interface, I'm -1. You don't need a PEP to define an
> > interface in Python - just establish an interface in the types you
> > care about. If you really mean to implement it primarily for some type
> > of yours, just do so (it' can't be *exactly* this interface, but you
> > can have the same operations, just in a different place).
> 
> That's what I have done, since I need it *now*, and in 2.2.

Ok, so this actually *is* an interface to some existing object. Which
one?

> I just did not want the PEP to collect too much dust: it should
> either be implemented or just die.

Then I would suggest that you withdraw it. To me, it sounds useless
unless significantly enhanced.

Regards,
Martin



From theller@python.net  Thu Dec 12 20:48:15 2002
From: theller@python.net (Thomas Heller)
Date: 12 Dec 2002 21:48:15 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <m3k7ifyno6.fsf@mira.informatik.hu-berlin.de>
References: <y96vaxwz.fsf@python.net>
 <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
 <bs3ratns.fsf@python.net>
 <m3of7ryor7.fsf@mira.informatik.hu-berlin.de>
 <znrb9dye.fsf@python.net>
 <m3k7ifyno6.fsf@mira.informatik.hu-berlin.de>
Message-ID: <u1hj9d68.fsf@python.net>

> > I just did not want the PEP to collect too much dust: it should
> > either be implemented or just die.
> 
> Then I would suggest that you withdraw it. To me, it sounds useless
> unless significantly enhanced.
> 

If there is still interest, it could be improved.
If not, it should better be withdrawn. I'll wait if
more responses com up.

Thanks for the review,

Thomas



From martin@v.loewis.de  Thu Dec 12 20:45:34 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 21:45:34 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <65tzasy5.fsf@python.net>
References: <y96vaxwz.fsf@python.net>
 <m31y4n109l.fsf@mira.informatik.hu-berlin.de>
 <65tzasy5.fsf@python.net>
Message-ID: <m3fzt3ynip.fsf@mira.informatik.hu-berlin.de>

Thomas Heller <theller@python.net> writes:

> Do you know my ctypes module? http://tinyurl.com/2tyx

Thanks for the pointer. I didn't know it; it looks quite powerful.

Regards,
Martin


From dave@boost-consulting.com  Thu Dec 12 21:09:09 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Thu, 12 Dec 2002 16:09:09 -0500
Subject: [Python-Dev] Possible low-hanging optimization
In-Reply-To: <15864.59369.517000.126606@montanaro.dyndns.org> (Skip
 Montanaro's message of "Thu, 12 Dec 2002 13:47:53 -0600")
References: <ud6o7une2.fsf@boost-consulting.com>
 <200212121807.gBCI7jL20642@odiug.zope.com>
 <15864.59369.517000.126606@montanaro.dyndns.org>
Message-ID: <u65tzndvu.fsf@boost-consulting.com>

Skip Montanaro <skip@pobox.com> writes:

>     Guido> For classic classes, it would be too much of a break with
>     Guido> history.
>
> In addition, since most classes exist to associate state with methods that
> operate on it you'd probably only be delaying the creation of self.__dict__
> by a few milliseconds in the common case.

That's why I mentioned that it might help for new-style classes with
__slots__.

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From guido@python.org  Thu Dec 12 21:57:45 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 16:57:45 -0500
Subject: [Python-Dev] Still want an expected skip file for regrtest.py?
In-Reply-To: Your message of "Wed, 11 Dec 2002 15:28:08 PST."
 <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU>
Message-ID: <200212122157.gBCLvjC02221@odiug.zope.com>

> I just updated my CVS copy of Python and noticed that regrtest.py was
> patched.  That triggered the memory of summarizing the whole expected skip
> thread from the last summary.  The conclusion to that thread was the idea
> of having a file that would list tests that could be added to  the
> expected skip  list at run-time and thus  squash the issue of having to
> ignore skipped test messages for things you knew would be skipped that
> were not listed on the platform skip list.
> 
> Well, with ``dummy_thread(ing)?`` sitting there on SF and knowing that I
> have a lot of PHP coding coming up, I am up for implementing this (with no
> specified timeframe).  The question is how do you want it implemented?  I
> say it should just be a file you keep in your Lib/test/ directory that is
> formatted like what you pass in for the -f option to regrtest.py (heck, I
> plan to copy the code that implements that option).  It should always be
> included without a passed-in option since the whole point of this is that
> these tests are expected to be skipped.  Thus it should have a
> standardized name (how about "ToSkip.txt"? Maybe a leading underscore?)

I leave the design mostly up to you.  I think it should use a file in
the current directory -- that's where you'd type "make test" anyway,
and when building multiple versions (e.g. a debug version or different
platforms) it may actually be useful to be able to have multiple
"skips" files.  (When using VPATH, that is.)

> The other issue is whether the listed tests should supplement or extend
> the expected tests for the platform.  I say it should extend, but the
> thread originally discussing this said supplement.  If both ways are
> desired, the file could specify either ``..add`` or ``..replace`` (or
> something similar) to signal which to do (default being to add them).  It
> could be required to be at the top of the file, at the end, or just
> anywhere; I have no preference.  But I think it should just extend it
> since we have those tests listed for a reason and having to possibly
> re-list them would be a pain.

I expect that the most common use would be for the file to augment the
built-in list.  Maybe entries with a leading minus could be deducted
from the built-in list.

> Or you could all just say "surprise me, Brett" and let me sweat the
> details.  =)

I'll say "amaze me, Brett." :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From martin@v.loewis.de  Thu Dec 12 22:46:17 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 12 Dec 2002 23:46:17 +0100
Subject: [Python-Dev] Tcl, Tkinter, and threads
In-Reply-To: <200212122021.gBCKL1T24010@odiug.zope.com>
References: <m3isxz14oy.fsf@mira.informatik.hu-berlin.de>
 <200212122021.gBCKL1T24010@odiug.zope.com>
Message-ID: <m37keevosm.fsf@mira.informatik.hu-berlin.de>

Guido van Rossum <guido@python.org> writes:

> Good work.  Thanks for taking over maintenance of _tkinter!

This reminds me that I forgot the acknowledgements in my previous
message:

I'd like to thank Matthias Klose of the Debian project for
coordinating the communication, and recherching various bits of
information, and Jeffrey Hobbs and Zoran Vasiljevic of the Tcl project
for giving valuable hints on how to approach this problem.

Regards,
Martin



From lars@gustaebel.de  Thu Dec 12 23:35:05 2002
From: lars@gustaebel.de (Lars =?iso-8859-1?q?Gust=E4bel?=)
Date: Fri, 13 Dec 2002 00:35:05 +0100
Subject: [Python-Dev] Tarfile support
In-Reply-To: <20021212182954.A30768@ibook.distro.conectiva>
References: <20021212112440.A25044@ibook.distro.conectiva> <200212122026.gBCKQ7x24337@odiug.zope.com> <20021212182954.A30768@ibook.distro.conectiva>
Message-ID: <200212130034.39814.lars@gustaebel.de>

> > No time to review the code, but the feature set and docs look very
> > good.
> >
> > Is there any chance that this can be contributed to the PSF under the
> > standard PSF license?
>
> Lars, can you please give us your position about this?

I must admit that tarfile.py's licensing does not really matter to me. If=
=20
changing its license to PSF makes people's life easier, it is absolutely=20
okay.

--=20
Lars Gust=E4bel
lars@gustaebel.de


From guido@python.org  Thu Dec 12 23:59:38 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 18:59:38 -0500
Subject: [Python-Dev] Tarfile support
In-Reply-To: Your message of "Fri, 13 Dec 2002 00:35:05 +0100."
 <200212130034.39814.lars@gustaebel.de>
References: <20021212112440.A25044@ibook.distro.conectiva> <200212122026.gBCKQ7x24337@odiug.zope.com> <20021212182954.A30768@ibook.distro.conectiva>
 <200212130034.39814.lars@gustaebel.de>
Message-ID: <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net>

> > > No time to review the code, but the feature set and docs look very
> > > good.
> > >
> > > Is there any chance that this can be contributed to the PSF under the
> > > standard PSF license?
> >
> > Lars, can you please give us your position about this?
> 
> I must admit that tarfile.py's licensing does not really matter to me. If 
> changing its license to PSF makes people's life easier, it is absolutely 
> okay.

Great!  I strongly prefer contributions under the standard PSF
license, so redistributors won't have to worry about separate
licensing conditions of individual components.

There's a draft contribution agreement at the PSF website:

  http://www.python.org/psf/psf-contributor-agreement.html

--Guido van Rossum (home page: http://www.python.org/~guido/)


From bac@OCF.Berkeley.EDU  Fri Dec 13 00:21:32 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Thu, 12 Dec 2002 16:21:32 -0800 (PST)
Subject: [Python-Dev] Still want an expected skip file for regrtest.py?
In-Reply-To: <200212122157.gBCLvjC02221@odiug.zope.com>
References: <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU>
 <200212122157.gBCLvjC02221@odiug.zope.com>
Message-ID: <Pine.SOL.4.50.0212121616580.22272-100000@death.OCF.Berkeley.EDU>

[Guido van Rossum]

> I leave the design mostly up to you.  I think it should use a file in
> the current directory -- that's where you'd type "make test" anyway,
> and when building multiple versions (e.g. a debug version or different
> platforms) it may actually be useful to be able to have multiple
> "skips" files.  (When using VPATH, that is.)
>

Using the current directory works for me.

As for multiple skips, I guess I could determine possible file names based
on the name of the platform ("ToSkipDARWIN.txt", "ToSkipDEBUG.txt", etc.).
Think I am going to have to read up on VPATH...

> I expect that the most common use would be for the file to augment the
> built-in list.  Maybe entries with a leading minus could be deducted
> from the built-in list.
>

Sounds reasonable.  I can go with that.  Might require tests to be on
individual lines (-f option just does a ``line.split()``).

> > Or you could all just say "surprise me, Brett" and let me sweat the
> > details.  =)
>
> I'll say "amaze me, Brett." :-)
>

Better not mess this up then.  =)

-Brett


From guido@python.org  Fri Dec 13 01:05:21 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Dec 2002 20:05:21 -0500
Subject: [Python-Dev] Still want an expected skip file for regrtest.py?
In-Reply-To: Your message of "Thu, 12 Dec 2002 16:21:32 PST."
 <Pine.SOL.4.50.0212121616580.22272-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU> <200212122157.gBCLvjC02221@odiug.zope.com>
 <Pine.SOL.4.50.0212121616580.22272-100000@death.OCF.Berkeley.EDU>
Message-ID: <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net>

> As for multiple skips, I guess I could determine possible file names based
> on the name of the platform ("ToSkipDARWIN.txt", "ToSkipDEBUG.txt", etc.).
> Think I am going to have to read up on VPATH...

No, you misunderstand.  When one builds for multiple platforms, each
platform has its own directory, which is the current directory when
you build (and test) for that platform, so the skip file can simply
have a fixed name in the current directory.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From brett@python.org  Fri Dec 13 01:58:18 2002
From: brett@python.org (Brett Cannon)
Date: Thu, 12 Dec 2002 17:58:18 -0800 (PST)
Subject: [Python-Dev] Still want an expected skip file for regrtest.py?
In-Reply-To: <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net>
References: <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU>
 <200212122157.gBCLvjC02221@odiug.zope.com>
 <Pine.SOL.4.50.0212121616580.22272-100000@death.OCF.Berkeley.EDU>
 <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <Pine.SOL.4.50.0212121754010.22272-100000@death.OCF.Berkeley.EDU>

[Guido van Rossum]

> > As for multiple skips, I guess I could determine possible file names based
> > on the name of the platform ("ToSkipDARWIN.txt", "ToSkipDEBUG.txt", etc.).
> > Think I am going to have to read up on VPATH...
>
> No, you misunderstand.  When one builds for multiple platforms, each
> platform has its own directory, which is the current directory when
> you build (and test) for that platform, so the skip file can simply
> have a fixed name in the current directory.
>

Ah, gotcha.  So then a single, fixed name will handle it since people can
just stick their file in the directories with the appropriate changes for
that platform.  Nifty.

Now I just need a good name.  ToSkip.txt?  SkipTests.txt?
TestsToSkip.txt?  Eh, not worth worrying about this exact moment.

-Brett



From martin@v.loewis.de  Fri Dec 13 09:05:28 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 13 Dec 2002 10:05:28 +0100
Subject: [Python-Dev] Still want an expected skip file for regrtest.py?
In-Reply-To: <Pine.SOL.4.50.0212121754010.22272-100000@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU>
 <200212122157.gBCLvjC02221@odiug.zope.com>
 <Pine.SOL.4.50.0212121616580.22272-100000@death.OCF.Berkeley.EDU>
 <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212121754010.22272-100000@death.OCF.Berkeley.EDU>
Message-ID: <m3fzt21e7b.fsf@mira.informatik.hu-berlin.de>

Brett Cannon <bac@OCF.Berkeley.EDU> writes:

> Now I just need a good name.  ToSkip.txt?  SkipTests.txt?
> TestsToSkip.txt?  Eh, not worth worrying about this exact moment.

The exact file name won't matter, but PleaseNoCamelNames. This is not
a Wiki :-)

Regards,
Martin


From mwh@python.net  Fri Dec 13 10:33:13 2002
From: mwh@python.net (Michael Hudson)
Date: 13 Dec 2002 10:33:13 +0000
Subject: [Python-Dev] Possible low-hanging optimization
In-Reply-To: David Abrahams's message of "Thu, 12 Dec 2002 16:09:09 -0500"
References: <ud6o7une2.fsf@boost-consulting.com> <200212121807.gBCI7jL20642@odiug.zope.com> <15864.59369.517000.126606@montanaro.dyndns.org> <u65tzndvu.fsf@boost-consulting.com>
Message-ID: <2mhedinr86.fsf@starship.python.net>

David Abrahams <dave@boost-consulting.com> writes:

> Skip Montanaro <skip@pobox.com> writes:
> 
> >     Guido> For classic classes, it would be too much of a break with
> >     Guido> history.
> >
> > In addition, since most classes exist to associate state with methods that
> > operate on it you'd probably only be delaying the creation of self.__dict__
> > by a few milliseconds in the common case.
> 
> That's why I mentioned that it might help for new-style classes with
> __slots__.

And why everyone seems to think we do it already in that case but
isn't quite sure :)  (I'm in that camp too).

Cheers,
M.

-- 
  Just point your web browser at http://www.python.org/search/ and
  look for "program", "doesn't", "work", or "my". Whenever you find
  someone else whose program didn't work, don't do what they
  did. Repeat as needed.    -- Tim Peters, on python-help, 16 Jun 1998


From lars@gustaebel.de  Fri Dec 13 11:34:19 2002
From: lars@gustaebel.de (Lars =?iso-8859-1?q?Gust=E4bel?=)
Date: Fri, 13 Dec 2002 12:34:19 +0100
Subject: [Python-Dev] Tarfile support
In-Reply-To: <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net>
References: <20021212112440.A25044@ibook.distro.conectiva> <200212130034.39814.lars@gustaebel.de> <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <200212131233.47430.lars@gustaebel.de>

> Great!  I strongly prefer contributions under the standard PSF
> license, so redistributors won't have to worry about separate
> licensing conditions of individual components.
>
> There's a draft contribution agreement at the PSF website:
>
>   http://www.python.org/psf/psf-contributor-agreement.html

Ok. If I understand all this correctly, the section entitled "Python=20
Contribution Licensing Agreement Version 1.0" is the appropriate one for =
this=20
particular case.
If this is right, the PSF will soon get a letter from me. Any hints on ho=
w to=20
describe the contribution accurately?

--=20
Lars Gust=E4bel
lars@gustaebel.de


From Jack.Jansen@cwi.nl  Fri Dec 13 12:33:16 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Fri, 13 Dec 2002 13:33:16 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <200212121910.gBCJAo222634@odiug.zope.com>
Message-ID: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl>

On Thursday, Dec 12, 2002, at 20:10 Europe/Amsterdam, Guido van Rossum 
wrote:
>> If you call YAGNI, it's ok for me, that's why I'm asking. Better save
>> the time in this case...
>
> Well, I have no personal need for it.  If you're the only one, that
> sounds like a big YAGNI to me...

Well... Note that you are currently living dangerously if you're using 
an s# format and access that buffer again after a 
Py_BEGIN_ALLOW_THREADS. If the object passed from Python was a mutable 
object with a buffer interface there's a chance that another thread has 
moved the underlying data.

The PyArg_Parse replacement I would like to do (for which I have had, 
unfortunately, absolutely no support up to now, and which I don't feel 
up to designing all by myself) could fix this with PEP298: it would use 
the locked buffer interface if available on the object, and release the 
locked buffer after your method returns.
--
Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma 
Goldman



From guido@python.org  Fri Dec 13 13:32:40 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 13 Dec 2002 08:32:40 -0500
Subject: [Python-Dev] Tarfile support
In-Reply-To: Your message of "Fri, 13 Dec 2002 12:34:19 +0100."
 <200212131233.47430.lars@gustaebel.de>
References: <20021212112440.A25044@ibook.distro.conectiva> <200212130034.39814.lars@gustaebel.de> <200212122359.gBCNxd021838@pcp02138704pcs.reston01.va.comcast.net>
 <200212131233.47430.lars@gustaebel.de>
Message-ID: <200212131332.gBDDWeF02668@pcp02138704pcs.reston01.va.comcast.net>

> > Great!  I strongly prefer contributions under the standard PSF
> > license, so redistributors won't have to worry about separate
> > licensing conditions of individual components.
> >
> > There's a draft contribution agreement at the PSF website:
> >
> >   http://www.python.org/psf/psf-contributor-agreement.html
> 
> Ok. If I understand all this correctly, the section entitled "Python
> Contribution Licensing Agreement Version 1.0" is the appropriate one
> for this particular case.  If this is right, the PSF will soon get a
> letter from me. Any hints on how to describe the contribution
> accurately?

Yup, that's the one.  I'd say "the tarfile.py module and its
documentation and unit tests" would cover your contribution
adequately.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From theller@python.net  Fri Dec 13 14:02:02 2002
From: theller@python.net (Thomas Heller)
Date: 13 Dec 2002 15:02:02 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl>
References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl>
Message-ID: <n0naow4l.fsf@python.net>

Jack Jansen <Jack.Jansen@cwi.nl> writes:

> On Thursday, Dec 12, 2002, at 20:10 Europe/Amsterdam, Guido van Rossum
> wrote:
> 
> >> If you call YAGNI, it's ok for me, that's why I'm asking. Better save
> >> the time in this case...
> >
> > Well, I have no personal need for it.  If you're the only one, that
> > sounds like a big YAGNI to me...
> 
> Well... Note that you are currently living dangerously if you're using
> an s# format and access that buffer again after a
> Py_BEGIN_ALLOW_THREADS. If the object passed from Python was a mutable
> object with a buffer interface there's a chance that another thread
> has moved the underlying data.

It can even by the same thread, executing an innocent Py_DECREF()
can execute arbitrary Python code. Thinking of it, it's not clear
what you are allowed to do after PyArg_Parse() if you still need
the 'parsed pointers'.

And that's the main point of the PEP: the 'old' buffer interface
is nearly useless if you want too be safe.

Thomas



From skip@pobox.com  Fri Dec 13 14:51:18 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 13 Dec 2002 08:51:18 -0600
Subject: [Python-Dev] Still want an expected skip file for regrtest.py?
In-Reply-To: <m3fzt21e7b.fsf@mira.informatik.hu-berlin.de>
References: <Pine.SOL.4.50.0212111455330.21072-100000@death.OCF.Berkeley.EDU>
 <200212122157.gBCLvjC02221@odiug.zope.com>
 <Pine.SOL.4.50.0212121616580.22272-100000@death.OCF.Berkeley.EDU>
 <200212130105.gBD15LI21955@pcp02138704pcs.reston01.va.comcast.net>
 <Pine.SOL.4.50.0212121754010.22272-100000@death.OCF.Berkeley.EDU>
 <m3fzt21e7b.fsf@mira.informatik.hu-berlin.de>
Message-ID: <15865.62438.953094.720317@montanaro.dyndns.org>

    Brett Cannon <bac@OCF.Berkeley.EDU> writes:
    > Now I just need a good name.  ToSkip.txt?  SkipTests.txt?
    > TestsToSkip.txt?  Eh, not worth worrying about this exact moment.

I'm always suspicious when things get named after me.  Will I have to
support them?  What about bretts_tests.txt? ;-)

Skip



From guido@python.org  Fri Dec 13 15:19:56 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 13 Dec 2002 10:19:56 -0500
Subject: [Python-Dev] PEP 298
In-Reply-To: Your message of "13 Dec 2002 15:02:02 +0100."
 <n0naow4l.fsf@python.net>
References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl>
 <n0naow4l.fsf@python.net>
Message-ID: <200212131519.gBDFJvM03149@odiug.zope.com>

> > Well... Note that you are currently living dangerously if you're using
> > an s# format and access that buffer again after a
> > Py_BEGIN_ALLOW_THREADS. If the object passed from Python was a mutable
> > object with a buffer interface there's a chance that another thread
> > has moved the underlying data.
> 
> It can even by the same thread, executing an innocent Py_DECREF()
> can execute arbitrary Python code. Thinking of it, it's not clear
> what you are allowed to do after PyArg_Parse() if you still need
> the 'parsed pointers'.
> 
> And that's the main point of the PEP: the 'old' buffer interface
> is nearly useless if you want too be safe.

Except that we can't force everyone to change at once, so we're stuck
with this...

Can you find an actual case where this could be used to cause a
segfault?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From theller@python.net  Fri Dec 13 15:28:56 2002
From: theller@python.net (Thomas Heller)
Date: 13 Dec 2002 16:28:56 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <200212131519.gBDFJvM03149@odiug.zope.com>
References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl>
 <n0naow4l.fsf@python.net> <200212131519.gBDFJvM03149@odiug.zope.com>
Message-ID: <65tyos3r.fsf@python.net>

> > And that's the main point of the PEP: the 'old' buffer interface
> > is nearly useless if you want too be safe.
> 
> Except that we can't force everyone to change at once, so we're stuck
> with this...
> 
> Can you find an actual case where this could be used to cause a
> segfault?

I'm not sure about core python, if one doesn't use the buffer object.
(Um, I probably shouldn't have mentioned this.)

And not anymore in my own extensions, but that's because I implemented
my own private scheme.

Thomas



From guido@python.org  Fri Dec 13 15:48:53 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 13 Dec 2002 10:48:53 -0500
Subject: [Python-Dev] PEP 298
In-Reply-To: Your message of "Fri, 13 Dec 2002 13:33:16 +0100."
 <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl>
References: <0DBCEFEB-0E97-11D7-A40E-0030655234CE@cwi.nl>
Message-ID: <200212131548.gBDFmrm09751@odiug.zope.com>

> The PyArg_Parse replacement I would like to do (for which I have had, 
> unfortunately, absolutely no support up to now, and which I don't feel 
> up to designing all by myself) could fix this with PEP298: it would use 
> the locked buffer interface if available on the object, and release the 
> locked buffer after your method returns.

I can't give you much more than moral support. :-(

It sounds like it would be a Big Change though; are you sure you're up
to this for Python 2.3?

--Guido van Rossum (home page: http://www.python.org/~guido/)



From jim@interet.com  Fri Dec 13 18:19:11 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 13 Dec 2002 13:19:11 -0500
Subject: [Python-Dev] new import hooks & zip import
References: <r01050400-1022-7C66986A0DBE11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <3DFA249F.20706@interet.com>

Just van Rossum wrote:
> After incorporating some suggestions and fixes from Paul Moore and Neal Norwitz
> and doing more testing myself, I've decided the importhook/zipimport patch is
> mature enough to post a proper patch:

In PEP 273 under "Booting" the directory /usr/local/lib/python23.zip
(for Unix) or python23.zip in the directory of python23.dll (for
Windows) was added to sys.path so that the Python standard library
could be loaded from a zip archive.  This is missing in your patch.

I am not sure whether this was deliberate.  But if you want to add
it, it requires patches to site.py, getpath.c, getpathp.c, pythonrun.c,
main.c and sysmodule.c.  This moves changes to sys.path out of
sysmodule.c and into getpath[p].c where they belong.  You may want to
use my patches for this even if you use your own import.c instead
of mine.

JimA



From just@letterror.com  Fri Dec 13 18:55:21 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 13 Dec 2002 19:55:21 +0100
Subject: [Python-Dev] new import hooks & zip import
In-Reply-To: <3DFA249F.20706@interet.com>
Message-ID: <r01050400-1022-71E000C70ECC11D7AD06003065D5E7E4@[10.0.0.23]>

James C. Ahlstrom wrote:

> In PEP 273 under "Booting" the directory /usr/local/lib/python23.zip
> (for Unix) or python23.zip in the directory of python23.dll (for
> Windows) was added to sys.path so that the Python standard library
> could be loaded from a zip archive.  This is missing in your patch.

True. I wasn't sure if there was consensus about the neccessity of this.

> I am not sure whether this was deliberate. 

I'm not against this at all, I've just been ignoring the issue ;-)

> But if you want to add
> it, it requires patches to site.py, getpath.c, getpathp.c, pythonrun.c,
> main.c and sysmodule.c.  This moves changes to sys.path out of
> sysmodule.c and into getpath[p].c where they belong. 

It might belong there, but I don't see an immediate reason why a default zip
entry on sys.path demands this reorganization as well.

> You may want to use my patches for this even if you use your own
> import.c instead of mine.

I'll go read them more closely, thanks.

Just


From jeremy@alum.mit.edu  Fri Dec 13 19:13:46 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Fri, 13 Dec 2002 14:13:46 -0500
Subject: [Python-Dev] warnings compiling tkinter
Message-ID: <15866.12650.879757.730361@slothrop.zope.com>

Should I be concerned about these warnings?  Perhaps my version of
Tcl/Tk is too old?

jeremy

gcc -g -Wall -Wstrict-prototypes -fPIC -DWITH_APPINIT=1 -DWITH_BLT=1 -I/usr/X11R6/include -I. -I/usr/home/jeremy/src/python/dist/src/./Include -I/usr/local/include -I/usr/home/jeremy/src/python/dist/src/Include -I/usr/home/jeremy/src/python/dist/src/build-pydebug -c /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c -o /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.o
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c: In function `SetVar':
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1486: warning: passing arg 7 of `var_invoke' from incompatible pointer type
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1496: warning: passing arg 8 of `var_invoke' from incompatible pointer type
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c: In function `GetVar':
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1537: warning: passing arg 6 of `var_invoke' from incompatible pointer type
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1537: warning: passing arg 7 of `var_invoke' from incompatible pointer type
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c: In function `UnsetVar':
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1565: warning: passing arg 6 of `var_invoke2' from incompatible pointer type
/usr/home/jeremy/src/python/dist/src/Modules/_tkinter.c:1565: warning: passing arg 7 of `var_invoke2' from incompatible pointer type
gcc -shared /usr/home/jeremy/src/python/dist/src/Modules/_tkinter.o /usr/home/jeremy/src/python/dist/src/Modules/tkappinit.o -L/usr/X11R6/lib -L/usr/local/lib -lBLT -ltk8.3 -ltcl8.3 -lX11 -o build/lib.linux-i686-2.3/_tkinter.so



From martin@v.loewis.de  Fri Dec 13 21:14:49 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 13 Dec 2002 22:14:49 +0100
Subject: [Python-Dev] warnings compiling tkinter
In-Reply-To: <15866.12650.879757.730361@slothrop.zope.com>
References: <15866.12650.879757.730361@slothrop.zope.com>
Message-ID: <m3bs3pwrhy.fsf@mira.informatik.hu-berlin.de>

Jeremy Hylton <jeremy@alum.mit.edu> writes:

> Should I be concerned about these warnings?  Perhaps my version of
> Tcl/Tk is too old?

They are probably const-correctness issues. The functions whose
addresses we are taking now expect const arguments, but didn't before.

Can you try the following patch? If that won't help, please report the
declarations of those functions in your Tcl version.

Regards,
Martin

Index: Modules/_tkinter.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Modules/_tkinter.c,v
retrieving revision 1.137
diff -u -r1.137 _tkinter.c
--- Modules/_tkinter.c	12 Dec 2002 19:05:48 -0000	1.137
+++ Modules/_tkinter.c	13 Dec 2002 21:09:27 -0000
@@ -57,6 +57,12 @@
 #include <tk.h>
 #endif
 
+/* For Tcl 8.2 and 8.3, CONST* is not defined. */
+#ifndef CONST84_RETURN
+#define CONST84_RETURN
+#define CONST
+#endif
+
 #define TKMAJORMINOR (TK_MAJOR_VERSION*1000 + TK_MINOR_VERSION)
 
 #if TKMAJORMINOR < 8002
@@ -1334,9 +1340,9 @@
 
 TCL_DECLARE_MUTEX(var_mutex)
 
-typedef const char* (*EventFunc1)(Tcl_Interp*, const char*, int);
-typedef const char* (*EventFunc2)(Tcl_Interp*, const char*, const char*, int);
-typedef const char* (*EventFunc3)(Tcl_Interp*, const char*, const char*, const char*, int);
+typedef CONST84_RETURN char* (*EventFunc1)(Tcl_Interp*, CONST char*, int);
+typedef CONST84_RETURN char* (*EventFunc2)(Tcl_Interp*, CONST char*, CONST char*, int);
+typedef CONST84_RETURN char* (*EventFunc3)(Tcl_Interp*, CONST char*, CONST char*, const char*, int);
 typedef struct VarEvent {
 	Tcl_Event ev; /* must be first */
 	TkappObject *self;
@@ -1456,9 +1462,9 @@
 
 static PyObject*
 var_invoke2(PyObject *_self, char* arg1, char* arg2, char* arg3, int flags,
-	   int (*func1)(Tcl_Interp*, const char*, int),
-	   int (*func2)(Tcl_Interp*, const char*, const char*, int),
-	   int (*func3)(Tcl_Interp*, const char*, const char*, const char*, int))
+	   int (*func1)(Tcl_Interp*, CONST char*, int),
+	   int (*func2)(Tcl_Interp*, CONST char*, CONST char*, int),
+	   int (*func3)(Tcl_Interp*, CONST char*, CONST char*, CONST char*, int))
 {
 	return var_invoke(_self, arg1, arg2, arg3, flags,
 			  (EventFunc1)func1, (EventFunc2)func2,



From arigo@tunes.org  Fri Dec 13 23:46:47 2002
From: arigo@tunes.org (Armin Rigo)
Date: Fri, 13 Dec 2002 15:46:47 -0800 (PST)
Subject: [Python-Dev] os.path.commonprefix
In-Reply-To: <15861.9426.521685.951820@montanaro.dyndns.org>; from skip@pobox.com on Mon, Dec 09, 2002 at 05:18:42PM -0600
References: <20021207160218.1528F49BA@bespin.org> <15858.8941.920830.141997@montanaro.dyndns.org> <20021209224815.8F7D94B52@bespin.org> <15861.9426.521685.951820@montanaro.dyndns.org>
Message-ID: <20021213234647.439FF2029@bespin.org>

Hello Skip,

On Mon, Dec 09, 2002 at 05:18:42PM -0600, Skip Montanaro wrote:
> How about just adding commonpathprefix?  As I mentioned before, there's an
> implementation in Tools/scripts/trace.py.

Yes, providing a semi-broken version for compatibility is something, but
providing the expected behavior too is much better (and clearly marks the
cases where the previous version must not be used, by providing an
alternative).

The trace.py implementation seems broken because it directly uses os.sep
instead of functions of os.path, ignoring os.altsep and possibly other
platform-specific issues.

> The docs for the function state, "Note that this may return invalid paths
> because it works a character at a time."  I suspect that's sufficient.

Indeed.


Armin



From goodger@python.org  Sat Dec 14 01:26:35 2002
From: goodger@python.org (David Goodger)
Date: Fri, 13 Dec 2002 20:26:35 -0500
Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects
 abstract.c,2.93.6.7,2.93.6.8
Message-ID: <BA1FF2F9.2D123%goodger@python.org>

This backport checkin (Oct 5 2002) changed the exception text between
2.2.1:

    >>> int(None)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: object can't be converted to int

and 2.2.2 (last line):

    >>> int(None)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: int() argument must be a string or a number

Should this change have been backported to 2.2.2 from the 2.3
codebase?

Patrick O'Brien brought this to my attention.  He found a test in the
Docutils test suite that broke under 2.2.2.  I've parameterized the
broken test to work on old & new Python versions, which I would have
had to do for Python 2.3 anyway.  While minor, I think this was an
enhancement and not a bug fix.

Reference: discussion in http://www.python.org/sf/563740

-- 
David Goodger  <goodger@python.org>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/



From xscottg@yahoo.com  Sat Dec 14 03:29:52 2002
From: xscottg@yahoo.com (Scott Gilbert)
Date: Fri, 13 Dec 2002 19:29:52 -0800 (PST)
Subject: [Python-Dev] PEP 298
In-Reply-To: <200212121910.gBCJAo222634@odiug.zope.com>
Message-ID: <20021214032952.63245.qmail@web40105.mail.yahoo.com>

Guido van Rossum wrote:
> There are doubts about its usefulness.  (AKA YAGNI.)  Would you also
> provide implementations for common object types?  Otherwise there
> wouldn't be a point, right?

Thomas Heller wrote:
> If you call YAGNI, it's ok for me, that's why I'm asking. Better save
> the time in this case...

This might be a YAGNI, since everyone can get by with there own private
strategy, but it's still a good idea.

Right now I have code that makes very specific checks on what type of
PyObject is passed in to a function to see whether it is "safe enough" to
release the GIL.  For instance, the current implementation of strings seems
to be safe enough for this, but most other buffer types aren't.  This is a
bummer since arrays aren't really "safe enough", and that's the kind of
data we (people at my company) usually want to pass to a C extension. 
Right now I tell them they have to do an array.tostring() call first. 
That's ugly because it makes a copy of a potentially huge chunk of memory.

It would be much better if an object could promise not to realloc or free
your pointer after the GIL is released by implementing this interface.

Martin v. Löwis wrote:
> This is flawed. According to the PEP, an object which has
> acquirelockedbuffer called will live forever:
> 
> # Retrieving a buffer from an object puts this object in a locked
> # state during which the buffer may not be freed, resized, or
> # reallocated.

The wording may not be specific enough, but the idea is that you can lock a
buffer object and should (must) unlock it later.  Between the lock and
unlock, you are promised that your pointer won't be reallocated.  You could
release the GIL so other threads can run concurrently on multi processor
hardware.  You could pass the pointer to an asynchronous I/O routine.  All
sorts of good stuff.

Martin v. Löwis wrote:
> If it is just the interface, I'm -1. You don't need a PEP to define an
> interface in Python - just establish an interface in the types you
> care about. 

There are several types I care about which are out of my control: string,
array, mmap, Numeric.array.


If it not getting into 2.3 implies that Thomas Heller doesn't want to work
on it, could you guys put it in the "abandoned" category instead of
"rejected" or "retracted".  It really is a good idea.




__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com


From martin@v.loewis.de  Sat Dec 14 07:56:30 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 14 Dec 2002 08:56:30 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <20021214032952.63245.qmail@web40105.mail.yahoo.com>
References: <20021214032952.63245.qmail@web40105.mail.yahoo.com>
Message-ID: <m3bs3p6nkh.fsf@mira.informatik.hu-berlin.de>

Scott Gilbert <xscottg@yahoo.com> writes:

> > # Retrieving a buffer from an object puts this object in a locked
> > # state during which the buffer may not be freed, resized, or
> > # reallocated.
> 
> The wording may not be specific enough, but the idea is that you can lock a
> buffer object and should (must) unlock it later.  

That is the flaw. It shouldn't tell me what I have to do, but instead
it should tell me what happens if I don't. If I don't unlock the
buffer, I would expect that it stays locked. Now it turns out that the
buffer can actually go away while being locked.

> > If it is just the interface, I'm -1. You don't need a PEP to define an
> > interface in Python - just establish an interface in the types you
> > care about. 
> 
> There are several types I care about which are out of my control: string,
> array, mmap, Numeric.array.

I see. So this is more than just the interface: your requirement is
that these types also implement it. Of course, Numeric.array is
somewhat out of control of the PEP.

If that is so, I'm -0: I still would like to see it become useful on
the Python level, i.e. the buffer object should somehow expose it.

Regards,
Martin



From xscottg@yahoo.com  Sat Dec 14 10:35:39 2002
From: xscottg@yahoo.com (Scott Gilbert)
Date: Sat, 14 Dec 2002 02:35:39 -0800 (PST)
Subject: [Python-Dev] PEP 298
In-Reply-To: <m3bs3p6nkh.fsf@mira.informatik.hu-berlin.de>
Message-ID: <20021214103539.9712.qmail@web40107.mail.yahoo.com>

--- "Martin v. Löwis" <martin@v.loewis.de> wrote:
>
> > > # Retrieving a buffer from an object puts this object in a locked
> > > # state during which the buffer may not be freed, resized, or
> > > # reallocated.
> > 
> > The wording may not be specific enough, but the idea is that you can
> > lock a buffer object and should (must) unlock it later.  
> 
> That is the flaw. It shouldn't tell me what I have to do, but instead
> it should tell me what happens if I don't.  If I don't unlock the buffer,

> I would expect that it stays locked.  Now it turns out that the buffer 
> can actually go away while being locked.
> 

It's not a flaw, and that's an odd criticism.  There aren't many functions
in the Python API that are documented in terms of what happens when you
misuse them.

Think of it as more like knowing when you have to call Py_INCREF and
Py_DECREF.  If you want to keep a pointer to an object, you call Py_INCREF.
 If you want to keep a pointer to a buffer after releasing the GIL, you
call PyObject_AcquireLockedWriteBuffer(...)  The normal use case would be:

     void* ptr; size_t len;
     PyObject_AcquireLockedBuffer(obj, &ptr, &len);
     Py_BEGIN_ALLOW_THREADS
     do_work_on(ptr);
     Py_END_ALLOW_THREADS
     PyObject_ReleaseLockedBuffer(obj);

If you think it's an issue, the act of locking the buffer could increase
the refcount too...

> > 
> > There are several types I care about which are out of my control:
> > string, array, mmap, Numeric.array.
> 
> I see. So this is more than just the interface: your requirement is
> that these types also implement it. Of course, Numeric.array is
> somewhat out of control of the PEP.
> 

Sure, the interface is only useful if people implement it.  If Python
defines the interface then we get interoperability instead of each package
inventing it's own scheme.

It's not unreasonable to assume that people would migrate to this over
time.  They're using the current buffer interface now, and many people are
aware of it's shortcomings.

>
> If that is so, I'm -0: I still would like to see it become useful on
> the Python level, i.e. the buffer object should somehow expose it.
> 

How would you use this at the Python level?  I don't see how it is useful
outside of a C extension.  I think it would be like exposing Py_INCREF and
Py_DECREF as Python builtins, but maybe you have a use case that I'm not
aware of.


Cheers,
    -Scott



__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com


From mal@lemburg.com  Sat Dec 14 11:03:57 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sat, 14 Dec 2002 12:03:57 +0100
Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects abstract.c,2.93.6.7,2.93.6.8
References: <BA1FF2F9.2D123%goodger@python.org>
Message-ID: <3DFB101D.6010306@lemburg.com>

David Goodger wrote:
> This backport checkin (Oct 5 2002) changed the exception text between
> 2.2.1:
> 
>     >>> int(None)
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>     TypeError: object can't be converted to int
> 
> and 2.2.2 (last line):
> 
>     >>> int(None)
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>     TypeError: int() argument must be a string or a number
> 
> Should this change have been backported to 2.2.2 from the 2.3
> codebase?
> 
> Patrick O'Brien brought this to my attention.  He found a test in the
> Docutils test suite that broke under 2.2.2. 

How can a test break when you change the error message text ?
I'd say that the test was broken: you should never
rely on a particular message text since these can
and do change rather often.

> I've parameterized the
> broken test to work on old & new Python versions, which I would have
> had to do for Python 2.3 anyway.  While minor, I think this was an
> enhancement and not a bug fix.
> 
> Reference: discussion in http://www.python.org/sf/563740
> 

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Sat Dec 14 11:43:49 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Sat, 14 Dec 2002 12:43:49 +0100
Subject: [Python-Dev] PEP 298
References: <20021214103539.9712.qmail@web40107.mail.yahoo.com>
Message-ID: <000701c2a366$122d3720$df2ae8d9@mira>

> It's not a flaw, and that's an odd criticism.  There aren't many
functions
> in the Python API that are documented in terms of what happens when
you
> misuse them.

There sure are. Looking at the sys module:
- exit: Most systems require it to be in the range 0-127, and produce
undefined results otherwise.
- _getframe: If that is deeper than the call stack, ValueError is
raised.
- setdefaultencoding: If name does not match any available encoding,
LookupError is raised.
and so on.

> Think of it as more like knowing when you have to call Py_INCREF and
> Py_DECREF.  If you want to keep a pointer to an object, you call
Py_INCREF.

Precisely, and this is my complaint. It ought to be like incref/decref,
but isn't. If I do incref without decref, I know what happens: the
object stays alive. If I do acquire without release, I would *expect*
that the the buffer stays acquired, but this is not necessarily the
case.

> If you think it's an issue, the act of locking the buffer could
increase
> the refcount too...

I know something is flawed. Whether this is fixed by changing the spec,
or by changing the implementation, I don't know.

If the spec means "the pointer is not valid if the reference count of
the underlying object goes to zero", it should say so. If the spec means
"if the reference count goes to zero while the object is locked, Python
aborts", it should say so. If the spec means "the buffer will not be
released while it is acquired", it should say so. From reading the spec,
I cannot tell which one it is. From reading the implementation, I can
tell "it depends on the type". I thought that "depends on the type" is
what this API was meant to eliminate.

> How would you use this at the Python level?  I don't see how it is
useful
> outside of a C extension.

I want to pass buffers to fcntl, and want fcntl to change the buffer
in-place. Currently, fcntl will need to keep the interpreter lock for
that.

For that to work, I need to create a locked buffer, and I need to find
out its address. I can create a buffer using the buffer builtin, but it
won't be a locked one, and I need to find out its address, but I can't.

Regards,
Martin



From goodger@python.org  Sat Dec 14 14:51:37 2002
From: goodger@python.org (David Goodger)
Date: Sat, 14 Dec 2002 09:51:37 -0500
Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects
 abstract.c,2.93.6.7,2.93.6.8
In-Reply-To: <3DFB101D.6010306@lemburg.com>
Message-ID: <BA20AFA8.2D13B%goodger@python.org>

M.-A. Lemburg wrote:
> How can a test break when you change the error message text ?
> I'd say that the test was broken: you should never
> rely on a particular message text since these can
> and do change rather often.

When Docutils reports errors, it sometimes uses the exception text returned
by Python.  The test that broke is one of those; it tests for an expected
exception.  I'm not questioning the validity of the change, just the
backport.  I'm fine with the change in 2.3, but I question whether the
changed behavior should have been back backported to 2.2.2.  I don't see the
old text as a bug; the new text is improved, true, but that says
"enhancement" to me, not "bugfix".  My post was a question that goes to
backport policy.

-- 
David Goodger  <goodger@python.org>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/



From pedronis@bluewin.ch  Sat Dec 14 15:29:27 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Sat, 14 Dec 2002 16:29:27 +0100
Subject: [Python-Dev] known obvious thing or bug (rexec)?
Message-ID: <015b01c2a385$97010700$6d94fea9@newmexico>

latest CVS

Python 2.3a0 (#1, Dec 14 2002, 07:19:32)
[GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

consider

>>> import rexec
>>> r=rexec.RExec()
>>> r.r_exec("class C: pass\nC.a=0")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/users/p/pe/pedronis/python/dist/src/Lib/rexec.py", line 309, in
r_
exec
    exec code in m.__dict__
  File "<string>", line 2, in ?
RuntimeError: classes are read-only in restricted mode

vs.

>>> r.r_exec("class C(object): pass\nC.a=0")
>>>

regards



From python@rcn.com  Sat Dec 14 16:52:39 2002
From: python@rcn.com (Raymond Hettinger)
Date: Sat, 14 Dec 2002 11:52:39 -0500
Subject: [Python-Dev] [Python-checkins] python/dist/src/Objectsabstract.c,2.93.6.7,2.93.6.8
References: <BA20AFA8.2D13B%goodger@python.org>
Message-ID: <002701c2a391$36de02e0$125ffea9@oemcomputer>

> When Docutils reports errors, it sometimes uses the exception text returned
> by Python.  The test that broke is one of those; it tests for an expected
> exception.  I'm not questioning the validity of the change, just the
> backport.  I'm fine with the change in 2.3, but I question whether the
> changed behavior should have been back backported to 2.2.2.  I don't see the
> old text as a bug; the new text is improved, true, but that says
> "enhancement" to me, not "bugfix".  My post was a question that goes to
> backport policy.

This particular rewording of the exception text was done for an 
instructor who found that the existing message was routinely 
confusing his students using Python.

It must be a point of view thing.  I think that usability problems are a type
of bug.  We often backport clarifications to doc strings, error messages,
and the LaTex documentation.  Using the same logic, GvR approved my
backporting changes to IDLE which were throwing novice users into
coniptions (like having RunScript enabled on the interactive screen).

I think the idea behind the no enhancement rule is to only
backport things learned by experience from the previously 
published version so that Py2.2.x becomes increasingly
perfected.   A genuine new feature has an experimental
quality in that it would not have gotten exercised in the previously 
published version.


From xscottg@yahoo.com  Sat Dec 14 18:04:43 2002
From: xscottg@yahoo.com (Scott Gilbert)
Date: Sat, 14 Dec 2002 10:04:43 -0800 (PST)
Subject: [Python-Dev] PEP 298
In-Reply-To: <000701c2a366$122d3720$df2ae8d9@mira>
Message-ID: <20021214180443.7819.qmail@web40104.mail.yahoo.com>

--- "Martin_v._Löwis" <martin@v.loewis.de> wrote:
> 
> > If you think it's an issue, the act of locking the buffer could
> > increase the refcount too...
> 
> I know something is flawed. Whether this is fixed by changing the spec,
> or by changing the implementation, I don't know.
> 
> If the spec means "the pointer is not valid if the reference count of
> the underlying object goes to zero", it should say so. If the spec means
> "if the reference count goes to zero while the object is locked, Python
> aborts", it should say so. If the spec means "the buffer will not be
> released while it is acquired", it should say so. From reading the spec,
> I cannot tell which one it is. From reading the implementation, I can
> tell "it depends on the type". I thought that "depends on the type" is
> what this API was meant to eliminate.
> 

The object should definitely be kept alive while someone has the buffer
locked.

I think it should be fixed by saying that PyObject_AcquireLockedReadBuffer
and PyObject_AcquireLockedWriteBuffer will also do a Py_INCREF and that
PyObject_ReleaseLockedBuffer will do a Py_DECREF.

If there is no disagreement, and Thomas Heller doesn't mind, I'll submit
the changes to the PEP...  Would this be sufficient to keep the PEP alive
(possibly in the "abandoned" state if Thomas Heller doesn't want to work on
it for 2.4)?

>
> > How would you use this at the Python level?  I don't see how it is
> > useful outside of a C extension.
> 
> I want to pass buffers to fcntl, and want fcntl to change the buffer
> in-place. Currently, fcntl will need to keep the interpreter lock for
> that.
> 
> For that to work, I need to create a locked buffer, and I need to find
> out its address. I can create a buffer using the buffer builtin, but it
> won't be a locked one, and I need to find out its address, but I can't.
> 

It sounds like you want to get your modifiable pointer as a Python integer.
 Then you want to pass that integer as the last argument of fcntl.  I think
it would be better if fcntl allowed the last argument to be a modifiable
buffer instead of just an "int or a string" as the docs indicate.

I think the fcntl C implementation could use the the locked buffer
interface when it is in place.  Then you could use the array module (or any
other modifiable buffer type object) for your argument.

BTW, the array object currently has the buffer_info() method which gives
you that integer.  I don't think this is the right long term solution, but
it might get you past a short term hump.



Cheers,
    -Scott








__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com


From jmiller@stsci.edu  Sat Dec 14 18:45:55 2002
From: jmiller@stsci.edu (Todd Miller)
Date: Sat, 14 Dec 2002 13:45:55 -0500
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
Message-ID: <3DFB7C63.4040401@stsci.edu>

I've been following the PEP-298 discussion a little, because I fear 
eventually IAGNI.   But when I look at the proposed changes,  I can't 
help but get the feeling that the "too wide" buffer interface just got 
wider.  By "too wide",  I am referring to the extraneous segmentation 
and character buffer aspects of the existing interface.  Adding three 
more calls for the locking problem bugs me.  Doesn't anything ever get 
simpler?

Looking at the proposed changes,  the only thing that really seems 
needed for the bufferprocs is the release call and a flag bit.  Assuming 
the release call is not implemented,  the acquired buffer is never 
locked or unlocked, and we're no worse off than we are now:  it's the 
BO's problem.   Assuming the release call is present,  the acquired 
buffer should *always* be locked and later released.   The existing 
wrapper functions (plus one for release) can hide/implement this, making 
it straight forward to provide correct extensions:  just implement a 
locking extension.  It seems to me that the existing (3!) calls for 
acquiring the buffer pointer can do just fine.   Except...

In the case of existing extensions,  there will certainly be no call to 
the release function.   So for an extension which is slow to update to 
the revised protocol,  there are negative consequences:  buffers which 
are never unlocked.   But so what?  These consequences seem less dire 
(if more probable) than the existing consequences of "moving buffers." 
 It would be a bigger problem if PEP-298 wound up fiddling with 
reference counts.  In that case,  un-released buffers would also become 
memory leaks.   But is it really necessary to mess with the reference 
counts?   I'm having a hard time imagining a situation in which the 
reference count for a buffer goes to zero while it is locked which is 
not a bug somewhere else.  What am I missing?  

Another possibility is to implement a "release buffer pointer" wrapper 
function which *does nothing* for a Python release 2.3 and change 
nothing else.   Let extensions which use buffers adopt usage of the 
wrapper and the locking protocol, with no immediate benefit.   Then 
implement locking for real in Python 2.4.   That way,  the number of 
non-conforming extensions can be minimized,  but the buffer API can be 
kept simple.  

So,  in the end,  what I'm wondering is whether there's a simpler middle 
ground between dropping the PEP and implementing it as it stands now.   
 Are there compromises to be found between guaranteeing correctness, 
backwards compatability, and growing complexity?   Is there another 
trade between backwards compatability and complexity which is tolerable 
for the short term with both simplicity and correctness available in the 
longer term?  

Todd




From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Sat Dec 14 18:52:01 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Sat, 14 Dec 2002 19:52:01 +0100
Subject: [Python-Dev] PEP 298
References: <20021214180443.7819.qmail@web40104.mail.yahoo.com>
Message-ID: <012001c2a3a1$e3a308d0$df2ae8d9@mira>

> It sounds like you want to get your modifiable pointer as a Python integer.
>  Then you want to pass that integer as the last argument of fcntl.  I think
> it would be better if fcntl allowed the last argument to be a modifiable
> buffer instead of just an "int or a string" as the docs indicate.

This is desirable, yes, and a patch is being implemented for that, see #555817. 
However, it is not sufficient: The /dev/poll interface on Solaris requires that
you must pass a structure that contains addresses to some storage.

> BTW, the array object currently has the buffer_info() method which gives
> you that integer.  I don't think this is the right long term solution, but
> it might get you past a short term hump.

Yes, Thomas just explained this to me, as well. I would have never expected it
to be at the array object, instead, I think it rightfully belongs to the buffer
objects. Having it at the buffer object also allows you get addresses of
other locked buffers - it would be unfortunate if we have the locked buffer interface,
but now you must copy string objects to array objects, just to get their address.

This is just as bad as having to copy array objects to string objects right now
to avoid relocations to the buffer.

Regards,
Martin



From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Sat Dec 14 19:16:13 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Sat, 14 Dec 2002 20:16:13 +0100
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFB7C63.4040401@stsci.edu>
Message-ID: <015201c2a3a5$44fb33c0$df2ae8d9@mira>

> I've been following the PEP-298 discussion a little, because I fear 
> eventually IAGNI.   But when I look at the proposed changes,  I can't 
> help but get the feeling that the "too wide" buffer interface just got 
> wider.  By "too wide",  I am referring to the extraneous segmentation 
> and character buffer aspects of the existing interface.  Adding three 
> more calls for the locking problem bugs me.  Doesn't anything ever get 
> simpler?

The locked buffer interface is simpler than the original buffer interface.
It does not support segmentation.

> It seems to me that the existing (3!) calls for 
> acquiring the buffer pointer can do just fine.   Except...
> 
> In the case of existing extensions,  there will certainly be no call to 
> the release function.

This is a minor problem. The major problem is that the existing protocol
does not guarantee locking. So if you find an object that supports the buffer
interface, you still don't know how long you can use the pointer 
you've received. It might become invalid after calling nearly any Python
API function.

The PEP authors believe that this can't be retroactively fixed 
by declaring the existing interfaces to be locked, and I tend to agree:
changing the docs doesn't change the code "out there".

> But is it really necessary to mess with the reference 
> counts?   I'm having a hard time imagining a situation in which the 
> reference count for a buffer goes to zero while it is locked which is 
> not a bug somewhere else.  What am I missing?  

Not sure what you are referring to here. If it is the PEP: it doesn't
tell whether reference counts should be increased. If you are referring
to my messages: I'm not saying that the reference counts should be increased.
I'm saying that the PEP does not specify what the interaction of locking
and reference counting is.

> So,  in the end,  what I'm wondering is whether there's a simpler middle 
> ground between dropping the PEP and implementing it as it stands now.   

There certainly is. The advantage of the PEP process is that there always
is a specific proposal to discuss. We only have this proposal at the moment,
so we can't discuss anything else. If you want to make a different proposal,
please propose specific wording (either as a new PEP, or as a modification
of the existing one). I can't picture precisely what it is that you are
proposing, nor what its consequences would be.

Regards,
Martin


From aahz@pythoncraft.com  Sat Dec 14 19:42:15 2002
From: aahz@pythoncraft.com (Aahz)
Date: Sat, 14 Dec 2002 14:42:15 -0500
Subject: [Python-Dev] Deprecate doctest?
In-Reply-To: <3DFB101D.6010306@lemburg.com>
References: <BA1FF2F9.2D123%goodger@python.org> <3DFB101D.6010306@lemburg.com>
Message-ID: <20021214194215.GA9536@panix.com>

On Sat, Dec 14, 2002, M.-A. Lemburg wrote:
> David Goodger wrote:
>>
>>This backport checkin (Oct 5 2002) changed the exception text between
>>2.2.1:
>>
>>    >>> int(None)
>>    Traceback (most recent call last):
>>      File "<stdin>", line 1, in ?
>>    TypeError: object can't be converted to int
>>
>>and 2.2.2 (last line):
>>
>>    >>> int(None)
>>    Traceback (most recent call last):
>>      File "<stdin>", line 1, in ?
>>    TypeError: int() argument must be a string or a number
>>
>>Should this change have been backported to 2.2.2 from the 2.3
>>codebase?
>>
>>Patrick O'Brien brought this to my attention.  He found a test in the
>>Docutils test suite that broke under 2.2.2. 
> 
> How can a test break when you change the error message text ? I'd
> say that the test was broken: you should never rely on a particular
> message text since these can and do change rather often.

That's tricky, given that doctest is in the standard library, and that's
exactly what doctest does.  Are you advocating deprecating doctest?
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From theller@python.net  Sat Dec 14 21:27:30 2002
From: theller@python.net (Thomas Heller)
Date: 14 Dec 2002 22:27:30 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <20021214180443.7819.qmail@web40104.mail.yahoo.com>
References: <20021214180443.7819.qmail@web40104.mail.yahoo.com>
Message-ID: <fzt0s33x.fsf@python.net>

Scott Gilbert <xscottg@yahoo.com> writes:

> --- "Martin_v._Löwis" <martin@v.loewis.de> wrote:
> > 
> > > If you think it's an issue, the act of locking the buffer could
> > > increase the refcount too...
> > 
> > I know something is flawed. Whether this is fixed by changing the spec,
> > or by changing the implementation, I don't know.

The implementation follows the PEP, of course, so the PEP must be
changed first. Currently the uploaded patch serves as a backup store
for the changes I've done so far.

> > 
> > If the spec means "the pointer is not valid if the reference count of
> > the underlying object goes to zero", it should say so. If the spec means
> > "if the reference count goes to zero while the object is locked, Python
> > aborts", it should say so. If the spec means "the buffer will not be
> > released while it is acquired", it should say so. From reading the spec,
> > I cannot tell which one it is. From reading the implementation, I can
> > tell "it depends on the type". I thought that "depends on the type" is
> > what this API was meant to eliminate.
> > 
> 
> The object should definitely be kept alive while someone has the buffer
> locked.
> 
> I think it should be fixed by saying that PyObject_AcquireLockedReadBuffer
> and PyObject_AcquireLockedWriteBuffer will also do a Py_INCREF and that
> PyObject_ReleaseLockedBuffer will do a Py_DECREF.
> 
> If there is no disagreement, and Thomas Heller doesn't mind, I'll submit
> the changes to the PEP...  Would this be sufficient to keep the PEP alive
> (possibly in the "abandoned" state if Thomas Heller doesn't want to work on
> it for 2.4)?

I will change the PEP according to Martin's 'complaints' and Scott's
suggestions probably on Monday, but feel free to do it yourself if you
want, Scott.

If I understand the PEP process correctly, the PEP is alive until
there is a definite pronouncement on it.

> > > How would you use this at the Python level?  I don't see how it is
> > > useful outside of a C extension.
> > 
> > I want to pass buffers to fcntl, and want fcntl to change the buffer
> > in-place. Currently, fcntl will need to keep the interpreter lock for
> > that.
> > 
> > For that to work, I need to create a locked buffer, and I need to find
> > out its address. I can create a buffer using the buffer builtin, but it
> > won't be a locked one, and I need to find out its address, but I can't.
> > 

A 'locked buffer' object could be built if the PEP is implemented,
should it be part of the PEP? But if you expose the buffer address to Python
as an integer, you're living dangerous again.

Thomas



From martin@v.loewis.de  Sat Dec 14 21:45:00 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 14 Dec 2002 22:45:00 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <fzt0s33x.fsf@python.net>
References: <20021214180443.7819.qmail@web40104.mail.yahoo.com>
 <fzt0s33x.fsf@python.net>
Message-ID: <m3smx0qnqb.fsf@mira.informatik.hu-berlin.de>

Thomas Heller <theller@python.net> writes:

> If I understand the PEP process correctly, the PEP is alive until
> there is a definite pronouncement on it.

That is certainly the case.

> A 'locked buffer' object could be built if the PEP is implemented,
> should it be part of the PEP? But if you expose the buffer address
> to Python as an integer, you're living dangerous again.

Why is that? Why does it add a danger that would not be there without
it.

It would be dangerous if you could create a buffer object by passing
an arbitrary integer, but I'm not proposing such a thing.

Regards,
Martin



From jmiller@stsci.edu  Sat Dec 14 22:44:31 2002
From: jmiller@stsci.edu (Todd Miller)
Date: Sat, 14 Dec 2002 17:44:31 -0500
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira>
Message-ID: <3DFBB44F.9070106@stsci.edu>

Martin v. Löwis wrote:

>>I've been following the PEP-298 discussion a little, because I fear 
>>eventually IAGNI.   But when I look at the proposed changes,  I can't 
>>help but get the feeling that the "too wide" buffer interface just got 
>>wider.  By "too wide",  I am referring to the extraneous segmentation 
>>and character buffer aspects of the existing interface.  Adding three 
>>more calls for the locking problem bugs me.  Doesn't anything ever get 
>>simpler?
>>    
>>
>
>The locked buffer interface is simpler than the original buffer interface.
>It does not support segmentation.
>  
>
I understand that,  it just seemed to me that the complexity of the 
PEP-298 solution is rising out of proportion with the problem: no locking.  

>  
>
>>It seems to me that the existing (3!) calls for 
>>acquiring the buffer pointer can do just fine.   Except...
>>
>>In the case of existing extensions,  there will certainly be no call to 
>>the release function.
>>    
>>
>
>This is a minor problem. The major problem is that the existing protocol
>does not guarantee locking. So if you find an object that supports the buffer
>interface, you still don't know how long you can use the pointer 
>you've received. It might become invalid after calling nearly any Python
>API function.
>
I'm suggesting that we should change the protocol to guarantee locking. 
  The consequences of doing that appear acceptable to me, and I was 
hoping to stir up some discussion on why they're *not*, or to simplify 
the PEP enough that the problem gets fixed rather than ignored.

>
>The PEP authors believe that this can't be retroactively fixed 
>by declaring the existing interfaces to be locked, and I tend to agree:
>changing the docs doesn't change the code "out there".
>  
>
I was trying to look at this from the perspective of what *really* 
breaks if we change the protocol to include locking implicitly.   If the 
consequences of "obsolescent extension code not calling the buffer 
release" are unnecessary exceptions,  that might be better than the 
current situation with no locking at all, where the consequences are a 
segfault or worse.   I'm also arguing that anywhere there are 
"consequences",  the current application is already in jeopardy  because 
it must be (a) using the buffer protocol and (b) attempting to modify 
the buffer address while the buffer is in use.   So,  either the 
application already has serious issues, or changing the protocol and the 
subsequent "dangling locks" won't hurt it at all.  Here I am assuming 
that locking does not affect refcount,  nor block deallocation.  (I'm 
also assuming that an object refcount cannot drop to zero during an 
extension function call w/o a bug somewhere else.  Help me out if I'm 
wrong here.)

>  
>
>>But is it really necessary to mess with the reference 
>>counts?   I'm having a hard time imagining a situation in which the 
>>reference count for a buffer goes to zero while it is locked which is 
>>not a bug somewhere else.  What am I missing?  
>>    
>>
>
>Not sure what you are referring to here. If it is the PEP: it doesn't
>tell whether reference counts should be increased. If you are referring
>to my messages: I'm not saying that the reference counts should be increased.
>I'm saying that the PEP does not specify what the interaction of locking
>and reference counting is.
>  
>
I was referring to the recent python-dev messages between yourself and 
Scott Gilbert.

>  
>
>>So,  in the end,  what I'm wondering is whether there's a simpler middle 
>>ground between dropping the PEP and implementing it as it stands now.   
>>    
>>
>
>There certainly is. The advantage of the PEP process is that there always
>is a specific proposal to discuss. We only have this proposal at the moment,
>so we can't discuss anything else. If you want to make a different proposal,
>please propose specific wording (either as a new PEP, or as a modification
>of the existing one). I can't picture precisely what it is that you are
>proposing, nor what its consequences would be.
>
Point taken.  What I'm really proposing is this:

1.  Add a buffer lock release call to the bufferprocs struct and an 
indicator flag bit.   Thomas's call is fine already.  The indicator bit 
could also remain the same,  but the new meaining would simply be that a 
"release" routine is supported.  The locking buffer pointer access calls 
should be dropped.

2.  Advise objects with support the buffer API to modify their *current* 
buffer pointer fetch routines to implement locking if they need it. 
 They must also fill in the release function pointer and set the bit to 
indicate they have it.  They need to modify their "mutator methods" to 
check the lock.

3. Add a buffer release wrapper function the C-API to hide the nasty 
details of checking for and calling the release function.  If there is 
none,  the release wrapper, is a no-op.   Thomas's existing release 
wrapper is fine.  The locking buffer pointer access calls should be dropped.

4. Advise extensions to treat all objects as implementing only a locking 
interface.  Get the locked pointer,   and release it later.   If the 
object chooses to ignore support for locking,  its the object's problem. 
 But there should be only *one* way to write an extension.  

5. Advise objects supporting the buffer API *not* to mess with refcounts 
during buffer protocol get/release.  This prevents probable (certain) 
memory leaks at the cost of improbable segfaults from obsolescent 
extensions.  Any extension that this is a problem for should be happy to 
have a solution and be willing to update to support locking.   I see no 
*new* risk here.

6. Advise objects which are asked to change a locked buffer to raise an 
exception rather than blocking.

7. Advise objects which are asked to deallocate while locked to 
deallocate normally.

>
>Regards,
>Martin
>  
>
Todd




From theller@python.net  Sat Dec 14 23:52:03 2002
From: theller@python.net (Thomas Heller)
Date: 15 Dec 2002 00:52:03 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <m3smx0qnqb.fsf@mira.informatik.hu-berlin.de>
References: <20021214180443.7819.qmail@web40104.mail.yahoo.com>
 <fzt0s33x.fsf@python.net>
 <m3smx0qnqb.fsf@mira.informatik.hu-berlin.de>
Message-ID: <7kecrwf0.fsf@python.net>

> > A 'locked buffer' object could be built if the PEP is implemented,
> > should it be part of the PEP? But if you expose the buffer address
> > to Python as an integer, you're living dangerous again.
> 
> Why is that? Why does it add a danger that would not be there without
> it.

Because you would pass the integer to a function expecting a pointer,
store it in a structure as a pointer, or something like that - what
else?  The danger is in the function, of course, but the integer
address probably encourages to implement those functions.

Thomas



From mmclay@comcast.net  Sun Dec 15 02:06:17 2002
From: mmclay@comcast.net (Michael McLay)
Date: Sat, 14 Dec 2002 21:06:17 -0500
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
Message-ID: <200212142106.01812.mclay@nist.gov>

sf/653938I is a patch to integrate the fixedpoint module, that was created by 
Tim Peters, into Python as a first class object. That is, the patch adds this 
new number type so that it has direct syntax support like float, int, long, 
str., etc. within the interpreter. 

I use Tim's module to implement the type. This limits the patch to a small bit 
of code that adds the a syntax interface to this module. Syntax recognition 
for the new number format required a change to Parser/tokenizer.c and to 
Python/compile.c.  This patch allows the new decimal type (I renamed the 
FixedPoint type to decimal in the fixedpoint.py file because the name is 
shorter and it is sufficient to distinguish the type from a binary float.) to 
be entered as a literal that is similar to a long, int, or float. The new 
syntax works as follows:

>>> 12.00d
12.00d
>>> .012d
0.012d
>>> 1d
1.d
>>> str(1.003d)
'1.003'

As you can see from the example, the default precision for decimal literals 
are determined by the precision entered expressed in the literal.

The patch also adds glue to Python/bltinmodule.c to create a builtin decimal() 
function. This builtin decimal (Formally FixedPoint) function invokes the 
constructor of the Python implementation of the decimal class defined in the 
fixedpoint module.  The implementation follows the familiar pattern for 
adding special builtin functions, much like the implementation of  apply and 
abs. It does not follow the typical method for adding built in types. That 
would require a more invasive patch. The builtin decimal function works as 
follows:

>>> decimal(1.333)
1.33d
>>> decimal(1.333,4)
1.3330d
>>> decimal(400,2)
400.00d

The semantics of the precision parameter may be incorrect because it is only 
addressing the precision of the fraction part of the number. Shouldn't it 
reflect the number of significant digits? If that were the case then the 
following would have been the result:

>>> decimal(400,2)
4.0e2d

This problem is more noticable when exponents are large relative to the number 
of significant digits. For instance:

>>> 40.e3d
40000.d

The result should have been 40.e3d, not 40000. This implies more precision 
than is declared by the constant.

As it currently is implemented the type pads with zeros on the integer side of 
the decimal point, whic implies more accuracy in the number than is true. I 
ran into this problem when I tried to implement the automatic conversion of 
string number values to decimal representation for numbers in 'e' format.

>>> 3.03e5d
303000.00d

The representation is incorrect. It should have returned 3.03e5d but it padded 
the result with bogus zeros.

For this first cut at a new decimal type I am primarily interested in 
investigating the semantics of the new native decimal number type in Python. 
This type will be useful to bankers, accountants, and newbies. The backend 
implementation could be replaced with a C implementation without affecting 
the Python language semantics. 

The approach used in this patch makes it very easy to experiment with changes 
to the semantics because all of the experimentation can be done by changing 
the code in Lib/fixedpoint.py. The compiled wrapper doesn't have to be 
modified and recompiled.

Unit testing so far just reuses the _test function in the fixedpoint.py 
module. Since I am not sure which way to go with the semantic interpretation 
of precision I decided to post the patch before making a significant change 
to fixedpoint.py. Some feedback on the interpretation of precision would be 
appreciated.

Documentation: 

The following needs to be added to section 2.1 of the library reference 
manual:

decimal(value, precision)
    Convert a string or a number to fixed point decimal number. If the 
argument is a string, it must contain a possibly signed decimal or floating 
point number, possibly embedded in whitespace. The precision parameter will 
be ignored on string values. The precision will be set based on the number of 
significant digits. Otherwise, the argument may be a plain or long integer or 
a floating point number, and a decimal number with the same value ( the 
precision attribute will set the precision of the fraction roundoff) is 
returned.


Section 3.1.1 of the tutorial will add a brief description of decimal number 
usage after the description of floating point: 

There is full support for floating point; operators with mixed type operands 
convert the integer operand to floating point:

>>> 3 * 3.75 / 1.5
7.5
>>> 7.0 / 2
3.5

Python also supports fixed point decimal numbers. These numbers do not suffer 
from the somewhat random roundoff errors that can occur with binary floating 
point numbers. A decimal number is created by adding a 'd' or 'D' suffix to a 
number:

>>> 3.3d + 3d
6.3d
>>> 3.3d + 3.03
6.3d
>>> 3.3d + decimal(3.03,3)
6.330d
>>> decimal(1.1, 16)
1.1000000000000001d
>>>
The builtin decimal constructor provides a means for converting float and int 
types to decimal types. Since floats are approximations of decimal floating 
point numbers there are often roundoff errors introduced by using floats.  
(The 1 in the last place of the conversion of 1.1 with 16 digits is a binary 
round off error.)  For this reason the decimal function requires the 
specification of precision when converting a float to a decimal. This allows 
the significant digits of the number to be specified. (Accountants and 
bankers love this because it allows them to balance the books without pennies 
being lost due to the use of binary numbers.)

>>> 3.3d/2
1.6d

 Note that in the example above the  expression 3.3d/2 returned 1.6d. The 
rounding scheme for Python decimals uses "banking" rounding rules. With 
floating point numbers the result would have been as follows:

>>> 3.3/2
1.6499999999999999

So as you can see the banking rules round off the .04999999999 portion of the 
number and calls it a an even 1.6.




From martin@v.loewis.de  Sun Dec 15 07:30:24 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 15 Dec 2002 08:30:24 +0100
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
In-Reply-To: <3DFBB44F.9070106@stsci.edu>
References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira>
 <3DFBB44F.9070106@stsci.edu>
Message-ID: <m37keboi27.fsf@mira.informatik.hu-berlin.de>

Todd Miller <jmiller@stsci.edu> writes:

> Point taken.  What I'm really proposing is this:

Thanks for detailing this proposal. I now agree with your assumption
that missing release calls might cause problems; I'd expect that they
indeed will cause problems in many applications.

The main issue is the interaction with ParseTuple, the 's', 'w', and
't' converters invoke a buffer operation, after which (under your
proposal) the object is locked, there is no corresponding release
call, and no place to put such a call.

So, once you have passed an array to some C function expecting a
char*, you cannot extend the array anymore.

Regards,
Martin


From martin@v.loewis.de  Sun Dec 15 07:33:06 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 15 Dec 2002 08:33:06 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <7kecrwf0.fsf@python.net>
References: <20021214180443.7819.qmail@web40104.mail.yahoo.com>
 <fzt0s33x.fsf@python.net>
 <m3smx0qnqb.fsf@mira.informatik.hu-berlin.de>
 <7kecrwf0.fsf@python.net>
Message-ID: <m33cozohxp.fsf@mira.informatik.hu-berlin.de>

Thomas Heller <theller@python.net> writes:

> > Why is that? Why does it add a danger that would not be there without
> > it.
> 
> Because you would pass the integer to a function expecting a pointer,
> store it in a structure as a pointer, or something like that - what
> else?  The danger is in the function, of course, but the integer
> address probably encourages to implement those functions.

But the functions are alreay there and implemented; as I said, I'm
concerned with the fcntl/ioctl family of functions. If "I can get the
address" would be encouraging to write functions that take arbitrary
integers as pointers, we would already have tons of such functions,
since the id() builtin also gives you an address.

Regards,
Martin



From brett@python.org  Sun Dec 15 09:00:33 2002
From: brett@python.org (Brett Cannon)
Date: Sun, 15 Dec 2002 01:00:33 -0800 (PST)
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212142106.01812.mclay@nist.gov>
References: <200212142106.01812.mclay@nist.gov>
Message-ID: <Pine.SOL.4.51.0212150055170.15168@death.OCF.Berkeley.EDU>

[Michael McLay]

> sf/653938I is a patch to integrate the fixedpoint module, that was created by
> Tim Peters, into Python as a first class object. That is, the patch adds this
> new number type so that it has direct syntax support like float, int, long,
> str., etc. within the interpreter.
>

Someone will correct me if I am wrong, but making a new decimal type that
includes a new suffix for numbers is  going to require a PEP.  Your
email, though, already seems like a rather good start for one and should
not require much more work.

It would be easier to get the module itself sans any Python core changes
accepted initially to gauge usage and interest by the Python community
before pushing for syntax integration.  This is what is be done with the
current rational implementation.

I personally would like to see the module included but currently say no to
any integration into the core syntax.

-Brett


From martin@v.loewis.de  Sun Dec 15 09:13:48 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 15 Dec 2002 10:13:48 +0100
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <Pine.SOL.4.51.0212150055170.15168@death.OCF.Berkeley.EDU>
References: <200212142106.01812.mclay@nist.gov>
 <Pine.SOL.4.51.0212150055170.15168@death.OCF.Berkeley.EDU>
Message-ID: <m3d6o3mypf.fsf@mira.informatik.hu-berlin.de>

Brett Cannon <bac@OCF.Berkeley.EDU> writes:

> Someone will correct me if I am wrong, but making a new decimal type that
> includes a new suffix for numbers is  going to require a PEP.  Your
> email, though, already seems like a rather good start for one and should
> not require much more work.

That is certainly the case (both the need for a PEP, and that the
message is a good starting point). While the technical aspects of the
proposed changes look fine, I'm missing a rationale: why is having
decimal literals in the language a good thing? [this is not to say
they aren't, but it may not be obvious]

> It would be easier to get the module itself sans any Python core changes
> accepted initially to gauge usage and interest by the Python community
> before pushing for syntax integration.  This is what is be done with the
> current rational implementation.

It also appears that a decimal type feature in the language will/should
interact with a rational type.

I don't know whether the current rational type proposals suggests
introduction of new literals, but I feel that the only rational
literals you'll ever need are decimals (and we all know that
floating-point literals are decimal rationals, too). So it might be
that we *only* get the decimal notation from this proposal, and not a
proper decimal type.

Of course, my CORBA affinity tells me that a decimal type would
be useful as well :-)

Regards,
Martin


From bsder@mail.allcaps.org  Sun Dec 15 09:54:56 2002
From: bsder@mail.allcaps.org (bsder@mail.allcaps.org)
Date: Sun, 15 Dec 2002 01:54:56 -0800 (PST)
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212142106.01812.mclay@nist.gov>
Message-ID: <Pine.LNX.4.44.0212150109110.16141-100000@mail.allcaps.org>

On Sat, 14 Dec 2002, Michael McLay wrote:

> sf/653938I is a patch to integrate the fixedpoint module, that was created by 

-1

Two reasons:

1) I don't see a PEP about this.  If I am wrong about this, please point 
me at the PEP so that I can review it.

2) There are official/semi-official standards for decimal floating-point
arithmetic. (IEEE 854 + ANSI X3.274 + ECMA 334)  The fixedpoint module
seems to comply with these standards only at the most basic levels.  
Again, if I am wrong, please point me at the relevant docs so that I can
review them.

-a



From fredrik@pythonware.com  Sun Dec 15 10:21:15 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Sun, 15 Dec 2002 11:21:15 +0100
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
References: <Pine.LNX.4.44.0212150109110.16141-100000@mail.allcaps.org>
Message-ID: <008f01c2a423$b7798400$ced241d5@hagrid>

> From: <bsder@mail.allcaps.org>
> 
> -1
>
> 1) I don't see a PEP about this.  If I am wrong about this, please point 
> me at the PEP so that I can review it.
> 
> 2) There are official/semi-official standards for decimal floating-point
> arithmetic. (IEEE 854 + ANSI X3.274 + ECMA 334)  The fixedpoint module
> seems to comply with these standards only at the most basic levels.  
> Again, if I am wrong, please point me at the relevant docs so that I can
> review them.
> 
> -a

arrogant anonymous python-dev voters appealing to authority?

interesting concept.

</F>



From skip@manatee.mojam.com  Sun Dec 15 13:00:16 2002
From: skip@manatee.mojam.com (Skip Montanaro)
Date: Sun, 15 Dec 2002 07:00:16 -0600
Subject: [Python-Dev] Weekly Python Bug/Patch Summary
Message-ID: <200212151300.gBFD0Gta017461@manatee.mojam.com>

Bug/Patch Summary
-----------------

321 open / 3121 total bugs (+4)
93 open / 1834 total patches (-1)

New Bugs
--------

email test external dependencies (2002-12-08)
	http://python.org/sf/650441
pydoc ignores $PAGER if TERM=='dumb' (2002-12-09)
	http://python.org/sf/651124
Review libshelve.tex when possible (2002-12-09)
	http://python.org/sf/651149
Bozo getstate w/ slots overrides base (2002-12-10)
	http://python.org/sf/651701
dynload_next needs better errors (2002-12-12)
	http://python.org/sf/652590
extra __builtin__ stuff not documented (2002-12-12)
	http://python.org/sf/652749
bad documentation for the "type" builtin (2002-12-12)
	http://python.org/sf/652888
py_compile does not return error code (2002-12-13)
	http://python.org/sf/653301
PyLong_AsVoidPtr()/PyLong_FromVoidPtr() (2002-12-13)
	http://python.org/sf/653542
Investigate weak linking (2002-12-14)
	http://python.org/sf/653871

New Patches
-----------

tarfile module implementation (2002-12-09)
	http://python.org/sf/651082
New import hooks + Import from Zip files (2002-12-12)
	http://python.org/sf/652586
PEP 298 implementation (2002-12-12)
	http://python.org/sf/652857
Generator form of os.path.walk (2002-12-12)
	http://python.org/sf/652980
Adds a builtin decimal type (FixedPoint) (2002-12-14)
	http://python.org/sf/653938

Closed Bugs
-----------

Fix the METH_CLASS implementation (2002-04-25)
	http://python.org/sf/548651
Tix Checklist getselection getstatus bug (2002-10-10)
	http://python.org/sf/621511
textwrap fails on unicode using defaults (2002-10-13)
	http://python.org/sf/622831
inconsistent results of leading whitespace in textwrap input (2002-10-13)
	http://python.org/sf/622849
xmlrpc can't do proxied HTTP (2002-12-04)
	http://python.org/sf/648658

Closed Patches
--------------

'switch'/'case'/'else' statement (2001-11-12)
	http://python.org/sf/481118
splitext performances improvement (2002-03-29)
	http://python.org/sf/536661
Add check for setrlimit() support (2002-05-03)
	http://python.org/sf/551960
Detect BLT on more systems (2002-10-26)
	http://python.org/sf/629126
tuple arg for issubclass (2002-12-06)
	http://python.org/sf/649608
avoid redefinition of pthread functions (2002-12-08)
	http://python.org/sf/650415


From jmiller@stsci.edu  Sun Dec 15 14:37:15 2002
From: jmiller@stsci.edu (Todd Miller)
Date: Sun, 15 Dec 2002 09:37:15 -0500
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira>	<3DFBB44F.9070106@stsci.edu> <m37keboi27.fsf@mira.informatik.hu-berlin.de>
Message-ID: <3DFC939B.2090609@stsci.edu>

Martin v. Löwis wrote:

>Todd Miller <jmiller@stsci.edu> writes:
>
>  
>
>>Point taken.  What I'm really proposing is this:
>>    
>>
>
>Thanks for detailing this proposal. I now agree with your assumption
>that missing release calls might cause problems; I'd expect that they
>indeed will cause problems in many applications.
>
Thanks for reading it.  

>
>The main issue is the interaction with ParseTuple, the 's', 'w', and
>'t' converters invoke a buffer operation, after which (under your
>proposal) the object is locked, there is no corresponding release
>call, and no place to put such a call.
>
I've used this aspect of PyArg_ParseTuple,  but forgot about it.    I 
was thinking that you're right,  the original object is "gone", we're 
stuck.   But...  it's still (in all probability) lying around in the arg 
tuple,  so some sort of PyArg_ReleaseBuffers is possible.   It could 
even be a near-carbon-copy of the PyArg_ParseTuple which "lost" the 
object to begin with,  to make the book keeping easy:

PyArg_ParseTuple(args, "iw#Os",  &someInt,  &writeableptr, 
&writeablesize, &someobject, &readableptr)

becomes:

PyArg_ReleaseBuffers(args, "iw#Os")

at exit time.   The parse spec defines which parts of the tuple to 
release, and which not to release.  Anything which is not w,s, or t gets 
skipped.  w,s,t are released.  With this call,  it becomes possible to 
"fix"  the PyArg_ParseTuple case.

Incidentally,  does PEP-298 solve this now?   (I think the answer is no. 
   PEP-298 is just using a different approach to "phased introduction" 
by leaving the original problem fully exposed in backwards compatability 
mode.  Or am I missing something here too?)

>
>So, once you have passed an array to some C function expecting a
>char*, you cannot extend the array anymore.
>
Well,  this does sound bad,  but I was assuming the core and library 
would be "fixed" at the first introduction of locking, and the only 
problem would be with non-conforming extensions.    I now think that 
"lock violations" might be better handled using the warning framework, 
defaulting to a one time warning.  

Todd





From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Sun Dec 15 15:47:26 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Sun, 15 Dec 2002 16:47:26 +0100
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFB7C63.4040401@stsci.edu> <015201c2a3a5$44fb33c0$df2ae8d9@mira>	<3DFBB44F.9070106@stsci.edu> <m37keboi27.fsf@mira.informatik.hu-berlin.de> <3DFC939B.2090609@stsci.edu>
Message-ID: <00c501c2a451$44cae8d0$d435e8d9@mira>

> I've used this aspect of PyArg_ParseTuple,  but forgot about it.    I 
> was thinking that you're right,  the original object is "gone", we're 
> stuck.   But...  it's still (in all probability) lying around in the arg 
> tuple,  so some sort of PyArg_ReleaseBuffers is possible.   

That looks clumsy, and also too easy to forget unless done systematically,
in which case it will add a considerable performance cost. I think the
locked buffer interface is much simpler in comparison.

If Jack ever manages to design an alternative parse tuple interface, I
assume it will arrange to automatically release the buffer when the function
returns (or, more precisely, when the buffer is deallocated.

In fact, my PEP 286 could address this issue: the locked buffers would be
associated with the enhanced tuple, so that releasing the tuple would unlock
the buffers. This *still* has the potential for breaking code, since now
the buffers are locked while the function runs (i.e. callbacks couldn't 
expand the array anymore); it is likely that such breakage would uncover
a bug in the locking assumptions of the existing code.

Your attempt to fix your proposal indicates one of the strengths of the PEP
process: most initial proposals have flaws, and, unless completely spelled
out, readers will typically miss the flaws on initial reading. The weakness,
of course, is that a PEP might be sitting around for a long time, and nobody
would be reviewing it.

> Incidentally,  does PEP-298 solve this now?

Depends on what "this" is. Existing code will continue to work as before:
No locking in ParseTuple happens, as a result, the arguments are unlocked
when the function returns; this is desirable. Since the buffers aren't 
locked, relocation may happen. If such relocation is ever observed, people
would have to replace the "s" parser with an "O" parser, and perform the
locking themselves.

So, yes, PEP-298 addresses this - although modification of existing code is
necessary to make use of this feature (as is in your fixed proposal).

> Well,  this does sound bad,  but I was assuming the core and library 
> would be "fixed" at the first introduction of locking, and the only 
> problem would be with non-conforming extensions.    I now think that 
> "lock violations" might be better handled using the warning framework, 
> defaulting to a one time warning.  

I think we should strive for a solution that keeps the function parameters
locked while the function runs, and unlocks them automatically at function 
return time.

This shows another flaw in the wording of PEP 298: It implies without saying
that the expectation is that you should have as many release calls as you
have lock calls. 

Regards,
Martin



From mmclay@comcast.net  Sun Dec 15 16:02:30 2002
From: mmclay@comcast.net (Michael McLay)
Date: Sun, 15 Dec 2002 11:02:30 -0500
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <Pine.SOL.4.51.0212150055170.15168@death.OCF.Berkeley.EDU>
References: <200212142106.01812.mclay@nist.gov>
 <Pine.SOL.4.51.0212150055170.15168@death.OCF.Berkeley.EDU>
Message-ID: <200212151102.30779.mmclay@comcast.net>

On Sunday 15 December 2002 04:00 am, Brett Cannon wrote:
> [Michael McLay]
>
> > sf/653938I is a patch to integrate the fixedpoint module, that was
> > created by Tim Peters, into Python as a first class object. That is, the
> > patch adds this new number type so that it has direct syntax support like
> > float, int, long, str., etc. within the interpreter.
>
> Someone will correct me if I am wrong, but making a new decimal type that
> includes a new suffix for numbers is  going to require a PEP.  Your
> email, though, already seems like a rather good start for one and should
> not require much more work.

Yes, there should be a PEP. I posted this as a work in progress because I had 
a question on the semantics and also to get a general sense of what questions 
a PEP might need to answer.
>
> It would be easier to get the module itself sans any Python core changes
> accepted initially to gauge usage and interest by the Python community
> before pushing for syntax integration.  This is what is be done with the
> current rational implementation.

The module has existed for quite some time and is available on sourceforge at 
<http://fixedpoint.sourceforge.net/>. I made some changes to the module and I 
am interested in opinions on the change in the name of the class. 

> I personally would like to see the module included but currently say no to
> any integration into the core syntax.

The syntax change is primarily to make Python more appealing to professions 
that require the semantics of fixed point decimal numbers. Ideally Python 
would only have decimal numbers. The use of binary floating point is a 
performance enhancement and since the language historically has used binary 
numbers for float the switch to using decimal numbers would break backward 
compatibllity. In order to add decimal numbers as an alternative the syntax 
must distinguish floats from decimal. The format I proposed is already used 
to distinguish number types is used for complex and long numbers.  

The syntax change would allow decimal numbers to be used without the clutter 
of a constructor and a quoted string. For example:

msql.execute ("INSERT INTO bankAccount
   VALUES ('%s','%s','%s','%s', (,'Pay Rent'
   ,1256,FixedPoint("1345.23") ,'1999-03-30',NULL)")

would be written as follows:

msql.execute ("INSERT INTO bankAccount
   VALUES ('%s','%s','%s','%s', (,'Pay Rent'
   ,1256,1345.23d ,'1999-03-30',NULL)")

In this case the percentage of clutter is pretty minimal, but if you want to 
use Python as a serious tool for interactive calculations the clutter becomes 
annoynig. For instance:

>>> def CalcAverage(list):
...     total = 0d
...     for i in list:
...        total += i
...     return total/len(list)
...
>>> 
CalcAverage([FixedPoint("123.40"),FixedPoint("222.20"),FixedPoint("300.33"),FixedPoint("400.32")])
FixedPoint("261.56")

Compare that to what would be required if the syntax were built into the 
language:

>>> CalcAverage([123.40d, 222.20d, 300.33d, 400.32d])
261.56d
>>>




From oren-py-d@hishome.net  Sun Dec 15 16:38:14 2002
From: oren-py-d@hishome.net (Oren Tirosh)
Date: Sun, 15 Dec 2002 11:38:14 -0500
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212151102.30779.mmclay@comcast.net>
References: <200212142106.01812.mclay@nist.gov> <Pine.SOL.4.51.0212150055170.15168@death.OCF.Berkeley.EDU> <200212151102.30779.mmclay@comcast.net>
Message-ID: <20021215163814.GA42886@hishome.net>

On Sun, Dec 15, 2002 at 11:02:30AM -0500, Michael McLay wrote:
> msql.execute ("INSERT INTO bankAccount
>    VALUES ('%s','%s','%s','%s', (,'Pay Rent'
>    ,1256,1345.23d ,'1999-03-30',NULL)")

I hope no financial software handling my money actually writes such
literals into the code. The above statement should not appear anywhere 
outside a tutorial.

A decimal literal might be handy when using the interactive Python prompt 
as a financial calculator, though. Is that enough justification for 
including it into the language? 

	Oren



From bbum@codefab.com  Sun Dec 15 17:58:28 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Sun, 15 Dec 2002 12:58:28 -0500
Subject: [Python-Dev] xmlrpc can't do proxied HTTP (2002-12-04)
In-Reply-To: <20021215170005.4432.76076.Mailman@mail.python.org>
Message-ID: <D0C8EA6A-1056-11D7-96EE-00306570BFF4@codefab.com>

On Sunday, Dec 15, 2002, at 12:00 US/Eastern, 
python-dev-request@python.org wrote:
> xmlrpc can't do proxied HTTP (2002-12-04)
> 	http://python.org/sf/648658

The above showed up in the 'closed' list.  I clicked through to see 
what was going on, but I received an 'invalid artifact ID' error.

?

b.bum



From jmiller@stsci.edu  Sun Dec 15 18:33:33 2002
From: jmiller@stsci.edu (Todd Miller)
Date: Sun, 15 Dec 2002 13:33:33 -0500
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
Message-ID: <3DFCCAFD.6020303@stsci.edu>


Martin v. Löwis wrote:

>>I've used this aspect of PyArg_ParseTuple,  but forgot about it.    I 
>>was thinking that you're right,  the original object is "gone", we're 
>>stuck.   But...  it's still (in all probability) lying around in the arg 
>>tuple,  so some sort of PyArg_ReleaseBuffers is possible.   
>>    
>>
>
>That looks clumsy, 
>
Well, yes, especially in light of PEP-286.  :)

>and also too easy to forget unless done systematically,
>
I really liked the idea of doing the buffer release behind the scenes, 
 as suggested by PEP-286 and below.

>in which case it will add a considerable performance cost. I think the
>
But won't *any* systematic solution have a performance cost?  Argtuples 
don't sound "free".  I'm really playing devil's advocate here,  because 
I like the concept of PEP-286.

>locked buffer interface is much simpler in comparison.
>
I disagree here.   I didn't see PEP-298 address PyArg_ParseTuple at all, 
 so there's no real basis for comparison on this point.    I'll grant 
you that PyArg_ReleaseBuffers is probably a lame solution to a problem 
better solved by PEP-286;  that said,  because use of 
PyArg_ReleaseBuffers is explicit,  there is no problem distributing the 
performance penalty to places where there is a corresponding benefit.  I 
was not suggesting dropping the existing release call;   merely adding a 
layer which could make sense out of PyArg_ParseTuple specs.

>
>If Jack ever manages to design an alternative parse tuple interface, I
>assume it will arrange to automatically release the buffer when the function
>returns (or, more precisely, when the buffer is deallocated.
>
>In fact, my PEP 286 could address this issue: the locked buffers would be
>associated with the enhanced tuple, so that releasing the tuple would unlock
>the buffers. 
>
That sounds pretty cool.

>This *still* has the potential for breaking code, since now
>the buffers are locked while the function runs (i.e. callbacks couldn't 
>expand the array anymore); it is likely that such breakage would uncover
>a bug in the locking assumptions of the existing code.
>  
>
I agree.  But the means exist to work around such problem (release and 
re-aquire),  and I would guess they are even rarer than the existing 
"buffer safety" issues.

>Your attempt to fix your proposal indicates one of the strengths of the PEP
>process: most initial proposals have flaws, and, unless completely spelled
>out, readers will typically miss the flaws on initial reading. The weakness,
>of course, is that a PEP might be sitting around for a long time, and nobody
>would be reviewing it.
>
>  
>
>>Incidentally,  does PEP-298 solve this now?
>>    
>>
>
>Depends on what "this" is. Existing code will continue to work as before:
>No locking in ParseTuple happens, as a result, the arguments are unlocked
>when the function returns; this is desirable. Since the buffers aren't 
>locked, relocation may happen. If such relocation is ever observed, people
>would have to replace the "s" parser with an "O" parser, and perform the
>locking themselves.
>
I'm just saying that we don't need 2 extra "locking calls" to achieve 
this.  

If the default behavior for the existing Python objects (array, mmap, 
...) is a onetime warning when a lock is abused,  where's the issue? 
 And with the warning framework,  the one time warning could be mutated 
into an exception or supressed by a variety of means.

My thinking is that *all* buffer pointers should be "logically" locked. 
  Objects that don't need locks don't need to change.   All extensions 
should be written and/or updated to handle objects which require 
locking.  Extensions which aren't updated will trigger warnings which 
could be supressed or converted into exceptions.

For me, that's simpler than saying "correct extensions" lock the buffer, 
 but "obsolete ones" may still segfault w/o warning.

Perhaps saying that "a warning is enough" is too limiting,  but this is 
a fringe issue,  so I think maybe it is.

>
>So, yes, PEP-298 addresses this - although modification of existing code is
>necessary to make use of this feature (as is in your fixed proposal).
>
>  
>
>>Well,  this does sound bad,  but I was assuming the core and library 
>>would be "fixed" at the first introduction of locking, and the only 
>>problem would be with non-conforming extensions.    I now think that 
>>"lock violations" might be better handled using the warning framework, 
>>defaulting to a one time warning.  
>>    
>>
>
>I think we should strive for a solution that keeps the function parameters
>locked while the function runs, and unlocks them automatically at function 
>return time.
>
I like this idea,  but wonder if it won't inflict a performance penalty 
on all function calls.

>
>This shows another flaw in the wording of PEP 298: It implies without saying
>that the expectation is that you should have as many release calls as you
>have lock calls. 
>
I was certainly assuming this was the case.

>
>Regards,
>Martin
>
>  
>
Regards,
Todd






From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Sun Dec 15 18:43:12 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Sun, 15 Dec 2002 19:43:12 +0100
Subject: [Python-Dev] xmlrpc can't do proxied HTTP (2002-12-04)
References: <D0C8EA6A-1056-11D7-96EE-00306570BFF4@codefab.com>
Message-ID: <016d01c2a469$d2b82500$d435e8d9@mira>

> The above showed up in the 'closed' list.  I clicked through to see 
> what was going on, but I received an 'invalid artifact ID' error.
> 
> ?

That's a buglet in both Skip's cronjob, and my SF redirector. It got 
moved to feature requests, and is now at

http://sourceforge.net/tracker/?group_id=5470&atid=355470&func=detail&aid=648658

Regards,
Martin




From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Sun Dec 15 19:09:23 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Sun, 15 Dec 2002 20:09:23 +0100
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFCCAFD.6020303@stsci.edu>
Message-ID: <017301c2a46d$7ade6160$d435e8d9@mira>

> I really liked the idea of doing the buffer release behind the scenes, 
>  as suggested by PEP-286 and below.

Alas, this has a fundamental flaw also, that's why I haven't advanced it,
yet. Jack found the flaw while reviewing it, but I did not manage to integrate
that into the PEP, and now I forgot what it was :-( It should be in the
mailing list archives, though.

> But won't *any* systematic solution have a performance cost?  Argtuples 
> don't sound "free".  I'm really playing devil's advocate here,  because 
> I like the concept of PEP-286.

Without any numbers to back up this claim: I believe the largest cost from
Py*Parse comes from the need to process the printf-style string, and the
corresponding va_arg processing, perhap along with the many type checks.

That's why I protest any unnecessary ParseTuple call, and object any
functions in this class. In comparison, the PEP 286 approach should
a) only produce costs if there is actually any buffer management, and
b) have a management processing cost (for building data structures and
   iterating over them) roughly the same size as the number of resources
   managed.

> I disagree here.   I didn't see PEP-298 address PyArg_ParseTuple at all, 
>  so there's no real basis for comparison on this point.    I'll grant 
> you that PyArg_ReleaseBuffers is probably a lame solution to a problem 
> better solved by PEP-286;  that said,  because use of 
> PyArg_ReleaseBuffers is explicit,  there is no problem distributing the 
> performance penalty to places where there is a corresponding benefit.  I 
> was not suggesting dropping the existing release call;   merely adding a 
> layer which could make sense out of PyArg_ParseTuple specs.

I understand that. But please compare

  if (!PyArg_ParseTuple(args,"sss",&string1, &string2, &string3))
     return NULL;
  if(!do_some_stuff(string1)){
     PyArg_Release(args,"sss",string1,string2,string3);
     return NULL;
  }
  if(!do_more_stuff(string2)){
     PyArg_Release(args,"sss",string1,string2,string3);
     return NULL;
  }
  compute_results(string3);

  PyArg_Release(args,"sss",string1,string2,string3);
  return result;

to

  if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3))
     return NULL;
  if(!do_some_stuff(string1)){
     return NULL;
  }
  if(!do_more_stuff(string2)){
     return NULL;
  }
  PyBuffer_Lock(obj3, &string3);
  compute_results(string3);
  PyBuffer_Unlock(obj3);

  return result;

In general, you won't have to lock everything, so you can avoid
a lot of clutter if you use just the PEP-298 interfaces, together
with an "O" parser.

> If the default behavior for the existing Python objects (array, mmap, 
> ...) is a onetime warning when a lock is abused,  where's the issue? 

It's unclear. I would expect that every application that uses arrays will
trigger the warning. When you get the warning, you will have no clue what
you did wrong, and most likely, the error is in the Python core. Who is
going to fix the hundreds and hundreds of functions that use the "s" parser,
and who is not going to be terribly bored by this, knowing that 99% of them
will never see an array.

> My thinking is that *all* buffer pointers should be "logically" locked. 

That is the intent of the PEP, indeed. For backwards compatibility, it
can't change the existing interface, but needs to create a new one. After
some time, the core could require that objects support it, much more later,
it could remove support for the old interface.

>   Objects that don't need locks don't need to change.   All extensions 
> should be written and/or updated to handle objects which require 
> locking.  Extensions which aren't updated will trigger warnings which 
> could be supressed or converted into exceptions.

The problem is that you are talking about virtually *all* extensions.
I bet you won't find a single extension that doesn't use an "s" parser.

> For me, that's simpler than saying "correct extensions" lock the buffer, 
>  but "obsolete ones" may still segfault w/o warning.

But, with your warning, this would still happen?

> I like this idea,  but wonder if it won't inflict a performance penalty 
> on all function calls.

I hope it won't. There is no implementation of enhanced argument tuples, yet
(and well may not appear for Python 2.3); but if this is done, a 
PyArg_ParseTuple call that doesn't allocate any resources will leave a NULL
pointer in the slot for the resources - on deallocation, all you pay is the
null pointer test.

Regards,
Martin



From lists@morpheus.demon.co.uk  Sun Dec 15 18:03:48 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Sun, 15 Dec 2002 18:03:48 +0000
Subject: [Python-Dev] Deprecate doctest?
References: <BA1FF2F9.2D123%goodger@python.org> <3DFB101D.6010306@lemburg.com>
 <20021214194215.GA9536@panix.com>
Message-ID: <n2m-g.7keb88hn.fsf@morpheus.demon.co.uk>

Aahz <aahz@pythoncraft.com> writes:

>> How can a test break when you change the error message text ? I'd
>> say that the test was broken: you should never rely on a particular
>> message text since these can and do change rather often.
>
> That's tricky, given that doctest is in the standard library, and
> that's exactly what doctest does.  Are you advocating deprecating
> doctest?

You can write the doctest to avoid the dependency on the exact
test. As a (none too realistic) example,

    >>> 1/0
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ZeroDivisionError: integer division or modulo by zero

is a bad test, because it depends on an exact text. (This one probably
isn't "bad" in practice, as I can't imagine the text changing very
often). On the other hand,

    >>> try:
    ...     1/0
    ... except ZeroDivisionError:
    ...     print "Divide by zero"
    ...
    Divide by zero

is a good test, because it protects against such a dependency. This is
a fairly well-known type of issue with doctest, which usually shows up
in the form of reliance on a particular order of printing a
dictionary. See section 5.2.7 of the manual (Warnings).

Paul.
-- 
This signature intentionally left blank


From jmiller@stsci.edu  Sun Dec 15 22:07:29 2002
From: jmiller@stsci.edu (Todd Miller)
Date: Sun, 15 Dec 2002 17:07:29 -0500
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira>
Message-ID: <3DFCFD21.2090707@stsci.edu>

Martin v. Löwis wrote:

>I understand that. But please compare
>
>  if (!PyArg_ParseTuple(args,"sss",&string1, &string2, &string3))
>     return NULL;
>  if(!do_some_stuff(string1)){
>     PyArg_Release(args,"sss",string1,string2,string3);
>     return NULL;
>  }
>  if(!do_more_stuff(string2)){
>     PyArg_Release(args,"sss",string1,string2,string3);
>     return NULL;
>  }
>  compute_results(string3);
>
>  PyArg_Release(args,"sss",string1,string2,string3);
>  return result;
>
>to  
>
>if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3))
>     return NULL;
>  if(!do_some_stuff(string1)){
>     return NULL;
>  }
>  if(!do_more_stuff(string2)){
>     return NULL;
>  }
>  PyBuffer_Lock(obj3, &string3);
>  compute_results(string3);
>  PyBuffer_Unlock(obj3);
>
>  return result
>
>In general, you won't have to lock everything, so you can avoid
>a lot of clutter if you use just the PEP-298 interfaces, together
>with an "O" parser.
>  
>
Well,   my initial proposal would have read more like:

 PyObject *result = NULL;

  if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3))
     goto fail_or_exit;
  if(!do_some_stuff(string1))
     goto fail_or_exit;
  if(!do_more_stuff(string2))
     goto fail_or_exit;

 result = compute_results(string3);

fail_or_exit:
  PyBuffer_Unlock(obj3);
  return result;

I'll concede that I'm not 100% comfortable with exit-gotos and that my 
implicit locking approach makes them necessary.

>>If the default behavior for the existing Python objects (array, mmap, 
>>...) is a onetime warning when a lock is abused,  where's the issue? 
>>    
>>
>
>It's unclear. I would expect that every application that uses arrays will
>trigger the warning. When you get the warning, you will have no clue what
>you did wrong, and most likely, the error is in the Python core. Who is
>
True.  Isolating the missing release call is hard.  Harder still will be 
isolating missing locks.

>going to fix the hundreds and hundreds of functions that use the "s" parser,
>and who is not going to be terribly bored by this, knowing that 99% of them
>will never see an array.
>  
>
I guess that's the beauty of PEP-286...   I now see that (or something 
like it) as the only possibility for really correcting the buffer problem. 

>  
>
>>My thinking is that *all* buffer pointers should be "logically" locked. 
>>    
>>
>
>That is the intent of the PEP, indeed. For backwards compatibility, it
>can't change the existing interface, but needs to create a new one. After
>some time, the core could require that objects support it, much more later,
>it could remove support for the old interface.
>  
>
I think that kind of backwards compatability just paves the wave for 
extensions to stay broken and complicates the object structures as well. 
  I guess PEP-298 does create the possibility for creating correct 
extensions,  I just hate the thought of 5 getpointer calls where there 
are now 3 and should be 2.

>>  Objects that don't need locks don't need to change.   All extensions
>>should be written and/or updated to handle objects which require 
>>locking.  Extensions which aren't updated will trigger warnings which 
>>could be supressed or converted into exceptions.
>>    
>>
>
>The problem is that you are talking about virtually *all* extensions.
>I bet you won't find a single extension that doesn't use an "s" parser.
>
Most of our (numarray) extensions don't use "s" parsers,  but I'll 
accept that it's not feasible to update all extensions.  

>>For me, that's simpler than saying "correct extensions" lock the buffer, 
>> but "obsolete ones" may still segfault w/o warning.
>>    
>>
>
>But, with your warning, this would still happen?
>  
>
Yes.  But it's a (minor) improvement because you have an immediate clue 
why the segfault occurred.

>  
>
>>I like this idea,  but wonder if it won't inflict a performance penalty 
>>on all function calls.
>>    
>>
>
>I hope it won't. There is no implementation of enhanced argument tuples, yet
>(and well may not appear for Python 2.3); but if this is done, a 
>PyArg_ParseTuple call that doesn't allocate any resources will leave a NULL
>pointer in the slot for the resources - on deallocation, all you pay is the
>null pointer test.
>  
>
Sounds good to me.  I hope you implement it!  If PEP-286 were here now, 
 a simplified PEP-298 would be possible.  Without PEP-286,  I doubt the 
buffer locking problem will ever be adequately fixed and life is going 
to get more complicated.  Fortunately for me,  it's all still 
theoretical...  

>Regards,
>Martin
>
>  
>
Regards,
Todd




From brett@python.org  Sun Dec 15 22:32:17 2002
From: brett@python.org (Brett Cannon)
Date: Sun, 15 Dec 2002 14:32:17 -0800 (PST)
Subject: [Python-Dev] Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212151102.30779.mmclay@comcast.net>
References: <200212142106.01812.mclay@nist.gov>
 <Pine.SOL.4.51.0212150055170.15168@death.OCF.Berkeley.EDU>
 <200212151102.30779.mmclay@comcast.net>
Message-ID: <Pine.SOL.4.51.0212151414290.22207@death.OCF.Berkeley.EDU>

[Michael McLay]

> On Sunday 15 December 2002 04:00 am, Brett Cannon wrote:
> > [Michael McLay]
> >
> > Someone will correct me if I am wrong, but making a new decimal type that
> > includes a new suffix for numbers is  going to require a PEP.  Your
> > email, though, already seems like a rather good start for one and should
> > not require much more work.
>
> Yes, there should be a PEP. I posted this as a work in progress because I had
> a question on the semantics and also to get a general sense of what questions
> a PEP might need to answer.
> >

OK.  But please do a PEP.  PEPs do not need to have  all the answers to
all questions.  You an have a "Considerations" section where you list
these questions with the pros and cons of possible decisions.  It will
also help to get more people to read it (i.e., people other than
python-dev).

> > It would be easier to get the module itself sans any Python core changes
> > accepted initially to gauge usage and interest by the Python community
> > before pushing for syntax integration.  This is what is be done with the
> > current rational implementation.
>
> The module has existed for quite some time and is available on sourceforge at
> <http://fixedpoint.sourceforge.net/>. I made some changes to the module and I
> am interested in opinions on the change in the name of the class.
>

It is still going to take someone to champion the module to get BDFL
pronouncment from Guido autonomously or get enough people to rally behind
to put pressure on Guido to let it in.

What  I would like to know is if Tim ever pushed to get the module in, and
if not, why?  Since he is the original author I assume he felt some reason
to write it (although it quite easily be "I was bored" <wink>) and yet
didn't feel like trying to get it into the stdlib.

>
> The syntax change is primarily to make Python more appealing to professions
> that require the semantics of fixed point decimal numbers.

I understand that, but who knows how many of those types  of  users are
out  there (current and potentiol).  Going so far as to integrate it into
Python requires proof that there is a demand.  Just saying it will help
make Python appealling to one specific group is not enough.  If  it was
Python would be diluted  with a ton of ideas that appeal to only a
specific group.

> compatibllity. In order to add decimal numbers as an alternative the syntax
> must distinguish floats from decimal. The format I proposed is already used
> to distinguish number types is used for complex and long numbers.
>

Understood and if  it is decided  to add decimals as a built-in I  think
the way you have  proposed works very well.

My advice is to champion getting the module into the language.  As of
right now that is  the key getting anything  to move forward.

-Brett


From martin@v.loewis.de  Sun Dec 15 23:57:02 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 16 Dec 2002 00:57:02 +0100
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
In-Reply-To: <3DFCFD21.2090707@stsci.edu>
References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira>
 <3DFCFD21.2090707@stsci.edu>
Message-ID: <m3fzsylttd.fsf@mira.informatik.hu-berlin.de>

Todd Miller <jmiller@stsci.edu> writes:

> I think that kind of backwards compatability just paves the wave for
> extensions to stay broken and complicates the object structures as
> well. I guess PEP-298 does create the possibility for creating correct
> extensions,  I just hate the thought of 5 getpointer calls where there
> are now 3 and should be 2.

I think this is the core of your objection. However, notice that all
types implementing have to be modified in either proposal, and that
adding the two getpointer functions should be trivial in most cases:
- if the type has read/write memory, implement them both the same
  way;
- if the type has read-only memory, set the write pointer to NULL
- make the getpointer function a wrapper around the existing
  getpointer, requesting only a single segment and simultaneously
  locking the buffer.

The overhead implementation-wise is really minimal; all the hard work
will come from the locking code - see Thomas' patch to the string
object for an example. We could even provide a Generic function which
delegates the new get functions to the existing ones, accessing
segment zero.

> Sounds good to me.  I hope you implement it!  If PEP-286 were here
> now, a simplified PEP-298 would be possible. 

I now recall what the issue is: you can't assume that modifying the
argument tuple object to record resources is feasible, as this breaks
apply (where in every case the user allocates a tuple, and where this
is a plain old tuple).

So I'm now envisioning a strategy where a thread-local
(i.e. thread-state associated) stack is maintained - on argument tuple
allocation, a frame is put on this stack, and all resources go into
that frame, on tuple decrefing, the frame is cleared. There would be
three kinds of resources: objects, void*, and locked buffers. For
efficiency, this stack and its frames would be a flat memory block.

In any case, I'm still -0 on PEP 298 as it stands: it should deal with
much more aspects of buffers (and I now consider argument parsing to
be one of them), when those things are added, I'll support the PEP -
your objection that this adds too much API is of no concern to me.

Regards,
Martin



From tim.one@comcast.net  Mon Dec 16 06:48:01 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Dec 2002 01:48:01 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212142106.01812.mclay@nist.gov>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEMIDCAB.tim.one@comcast.net>

[Michael McLay]
> ...
> The semantics of the precision parameter may be incorrect because
> it is only addressing the precision of the fraction part of the
> number.

That's why the module was called FixedPoint.

> Shouldn't it reflect the number of significant digits?

That would be more appropriate to a floating point scheme.  Aahz has been
working on a Python implementation of Mike Cowlishaw's hybrid proposal:

    http://www2.hursley.ibm.com/decimal/

for about, oh, fifteen years now <wink>.

> ...
> >>> 3.03e5d
> 303000.00d
>
> The representation is incorrect. It should have returned 3.03e5d
> but it padded the result with bogus zeros.

It seems you don't really want a fixed-point type.  The FixedPoint
constructor avoided this rat hole by making the user be explicit about the
fixed number of fractional digits (or accept the default 2).

> ...
> Python also supports fixed point decimal numbers. These numbers
> do not suffer from the somewhat random roundoff errors that can occur
> with binary floating point numbers.

Yes they do.  In fact, roundoff error in decimal arithmetic is actually a
little *worse* than in binary arithmetic.  The saving grace of decimal
arithmetic has to do with representation error, not with roundoff error:
decimal numbers are exactly representable in a decimal arithmetic system,
but "almost all" decimal numbers can't be represented exactly in a binary
arithmetic system.  In addition, under FixedPoint's unusual "unbounded
before the decimal point, fixed # of digits after, and when mixing precision
the result gets the larger of the input precisions" rules, all of the
following are always exact operations:

    to_decimal_string
    from_decimal_string
    + - *
    mod
    divmod

Take away the "unbounded before the decimal point" and/or the "result
inherits the larger input precision" rules and the arithmetic operations
suffer roundoff errors, just as bad (actually a little worse) as binary
fixed-#-of-bits fp suffers.

> ...
>>> 3.3d + decimal(3.03,3)
> 6.330d
> >>> decimal(1.1, 16)
> 1.1000000000000001d
> >>>
> The builtin decimal constructor provides a means for converting
> float and int types to decimal types. Since floats are approximations
> of  decimal floating point numbers there are often roundoff errors
> introduced by using floats.
> (The 1 in the last place of the conversion of 1.1 with 16 digits
> is a binary round off error.)

That's really representation error.  Until the distinction becomes clear,
you'll keep believing that decimal arithmetic is somehow better <wink>.

[martin@v.loewis.de]
> ...
> It also appears that a decimal type feature in the language
> will/should interact with a rational type.

Possibly.  This is easy for FixedPoint, becauase internally they're actually
unbounded rationals in unnormalized form, with the denominator constrained
to be an exact power of 10.

[bsder@mail.allcaps.org]
> ...
> 2) There are official/semi-official standards for decimal floating-
>    point arithmetic. (IEEE 854 + ANSI X3.274 + ECMA 334)  The
>    fixedpoint module seems to comply with these standards only at the
>    most basic levels.

To the extent that it complies with them at all, that's an accident.  See
above:  the overwhelming design goal of FixedPoint was to eliminate rounding
error entirely in the basic arithmetic operations.  That can't be achieved
for division, but can be (and was) for the rest.  Division settled for
as-if-to-infinite-precision single-rounding to-nearest/even, which is the
least error-prone way to do it when you *have* to lose information.

[Brett Cannon]
> ...
> It is still going to take someone to champion the module to get BDFL
> pronouncment from Guido autonomously or get enough people to rally
> behind to put pressure on Guido to let it in.

Guido already agreed to fold FixedPoint into 2.3, provided someone other
than me do the work <wink>.  Doug Fort has done a great job on it (including
LaTeX docs, Fred!), and we're just missing someone with the time/desire to
do final integration.

That's independent of Michael's patch, of course.  The FixedPoint module is
just another library module.

> What I would like to know is if Tim ever pushed to get the module in,
> and if not, why?

I've never pushed for any of my modules to get in.  If a module attracts
enough users, it eventually becomes clear that it should go in, else it
becomes clear that it should remain out.  FixedPoint has had a small but
indecently grateful <wink> and growing audience for several years now.

> Since he is the original author I assume he felt some reason
> to write it (although it quite easily be "I was bored" <wink>) and yet
> didn't feel like trying to get it into the stdlib.

At the time I wrote it, numerically incorrect "fixed point" / "BCD" /
"decimal" classes were common on c.l.py.  Simulating decimal arithmetic
correctly isn't difficult if you've got the right background for it, but
most people who want it don't have the right background to implement it.
OTOH, I had a crushingly strong background to implement it, but didn't want
it for myself.  So after years of pointing out errors in flawed modules of
this kind, it dawned on me that I could save more time overall by writing a
correct implementation, as surprise-free as numerically possible.  That's
all there was to it, and you haven't seen a bad decimal module on c.l.py
ever since <wink>.

If Python were to grow a native decimal type, I'd rather move to Cowlishaw's
design, because it has the best shot at becoming a standard influential
enough to sway HW design (decimal fp in HW wouldn't be notably more
expensive or slower than binary fp in HW, given current chip capacities;
this was a much bigger deal when chip real estate was scarce).

>> The syntax change is primarily to make Python more appealing to
>> professions that require the semantics of fixed point decimal numbers.

> I understand that, but who knows how many of those types of users are
> out there (current and potentiol).

"Do applications actually use decimal data?" from Cowlishaw's FAQ should go
a way toward answering that:

    http://www2.hursley.ibm.com/decimal/decifaq.html



From brett@python.org  Mon Dec 16 07:07:04 2002
From: brett@python.org (Brett Cannon)
Date: Sun, 15 Dec 2002 23:07:04 -0800 (PST)
Subject: [Python-Dev] summary for 2002-12-01 through 2002-12-15
Message-ID: <Pine.SOL.4.51.0212152302580.6942@death.OCF.Berkeley.EDU>

Yes, it's that time again!  You guys (are there any ladies on python-dev?)
have about a day to let me know about my mistakes before I send this out
to the rest of the world.

Oh, I am leaving off the FixedPoint discussion on this summary on purpose;
I feel that it will be all clarified soon and so I am going to wait to
summarize it after that has occurred.

And Martin, your name is correctly spelled (complete with emulaut!) in the
original and should hopefully show up when I send it out to c.l.p.

Enjoy.

+++++++++++++++++++++++++++++++++++++++++++++++++++++
python-dev Summary for 2002-12-01 through 2002-12-15
+++++++++++++++++++++++++++++++++++++++++++++++++++++

======================
Summary Announcements
======================

Aahz  suggested that I try writing the summary from a third-person
perspective so as to remove ambiguity if anyone ever quotes something from
a summary which was written in first-person.  I am giving it a go in this
summary although there is not much here that is affected.

There was an error in the last summary where I gave Guido's position on
something incorrectly (summary of dict evaluation order).  It has been
fixed.  I am going to fix errors like that in old summaries with an errata
note in the text version.  It won't show up in the HTML version, though.

And go to PyCon_.  =)

.. _PyCon: http://www.python.org/pycon/

===================================
`New universal import mechanism`__
===================================
__ http://mail.python.org/pipermail/python-dev/2002-December/030602.html

Splinter threads:
	- `New and Improved Import Hooks
<http://mail.python.org/pipermail/python-dev/2002-December/030638.html>`__
	- `Re: New and Improved Import Hooks
<http://mail.python.org/pipermail/python-dev/2002-December/030708.html>`__
	- `Re[2]: [Python-Dev] New and Improved Import Hooks
<http://mail.python.org/pipermail/python-dev/2002-December/030776.html>`__
	- `New Import Hook & New Zip Importer
<http://mail.python.org/pipermail/python-dev/2002-December/030755.html>`__
	- `(no subject)
<http://mail.python.org/pipermail/python-dev/2002-December/030812.html>`__
	- `Another approach for the import mechanism
<http://mail.python.org/pipermail/python-dev/2002-December/030828.html>`__
	- `Re: Another approach for the import mechanism
<http://mail.python.org/pipermail/python-dev/2002-December/030985.html>`__
	- `New Import Hook & New Zip Importer
<http://mail.python.org/pipermail/python-dev/2002-December/030755.html>`__
	- `Import: where's the PEP?
<http://mail.python.org/pipermail/python-dev/2002-December/030868.html>`__
	- `zipimport & import hooks
<http://mail.python.org/pipermail/python-dev/2002-December/030886.html>`__
	- `VFS
<http://mail.python.org/pipermail/python-dev/2002-December/030952.html>`__
	- `Zip imports, PEP 273, and the .zip extension
<http://mail.python.org/pipermail/python-dev/2002-December/030953.html>`__
	- `Import compromise
<http://mail.python.org/pipermail/python-dev/2002-December/030956.html>`__
	- `zipimport, round 3 (or would that be 37?)
<http://mail.python.org/pipermail/python-dev/2002-December/030968.html>`__
	- `Complexity of import.c
<http://mail.python.org/pipermail/python-dev/2002-December/031000.html>`__
	- `zipimport, round 4
<http://mail.python.org/pipermail/python-dev/2002-December/031009.html>`__
	- `new import hooks & zip import
<http://mail.python.org/pipermail/python-dev/2002-December/030755.html>`__

In case you can't figure it out from the title of all of the splnter
threads, there was an immense discussion about `PEP 273`_ and getting an
import mechanism for zipped modules.  To give a little back-story, PEP 273
proposes allowing modules to be put into a zip file for easy distribution.
You could then just add the zip file to ```sys.path`_`` , PYTHONPATH_ ,
etc. and importing from the zip file would be handled properly.  It was
scheduled to be included in Python 2.3 thanks to Paul Moore keeping James
Ahlstrom's patch synced with CVS.  All was right with the world, more or
less.

But then Wiktor Sadowski commented on a checkin for `import.c`_ and how he
would like a better way to write custom imports.  This led to Just van
Rossum to write a new import mechanism.  Basically Just was unhappy with
how much fiddling the PEP 273 patch was doing to import.c.  So, just like
any self-respecting programmer would do, he came up with his own solution.
=)  Most of the discussion revolved around a few central points.

One was whether this was even worth the effort.  Since the PEP 273 patch
worked, why should Just bother reinventing the wheel?  The whole reason
Just decided to come up with another implementation was a major point.
The other was that he felt he could come up with a more general way of
allowing custom importers *and* have zip imports a part of the core.
There was also a discussion of whether Gordon McMillan's `iu.py`_ wasn't a
good solution; it was already well-tested and coded.  But Just wanted a
certain functionality that iu.py didn't offer...

Just wanted to be able to put arbitrary objects in ``sys.path``.  This
would allow the object to be called to figure out whether it had the
module being requested or not.  Taking the zip importer as an example,
there would be a zipimporter instance in ``sys.path`` for each zip file in
the path (once it was discovered it was a zip file; bootstrapping has been
carefully handled).  The issue of backward-compatibility was instantly
brought up.  It has been assumed that everything in ``sys.path`` was a
string and starting to include zipimporter objects would break that.  The
idea of making zipimporters (and any subsequent importers in the stdlib)
subclasses of strings was brought up and basically accepted, although
Guido viewed it as an ugly hack.

But in the end, Just's version acts like iu.py.  The basic algorithm for
handling zip files is::

    def find_module(name, path):
        if isbuiltin(name):
            return builtin_filedescr
        if isfrozen(name):
            return frozen_filedescr
        if path is None:
            path = sys.path
        for p in sys.path:  # I think Just meant to say ``path`` here
            try:
                v = zipimporter(p)
            except ZipImportError:
                pass
            else:
                w = v.find_module(name)
                if w is not None:
                    return w
            ...handle builtin file system import...

It is easily generalized to work for any registered importers.  There is
also the idea of adding something called a "metapath" that iu.py has.  It
would allow one to override any import by catching it before trying
``sys.path``.

One other discussion was over this was going to need a PEP.  Just has
written up an explanation, but as to whether a PEP is required has not
been decided.

Just's implementation is now a patch at http://www.python.org/sf/652586 .

.. _PEP 273: http://www.python.org/peps/pep-0273.html
.. _sys.path: http://www.python.org/doc/current/lib/module-sys.html
.. _PYTHONPATH: http://www.python.org/doc/current/tut/node8.html
.. _import.c:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Python/compile.c
.. _iu.py: http://www.mcmillan-inc.com/iu.html

===================
`Mac OSX issues`__
===================
__ http://mail.python.org/pipermail/python-dev/2002-December/030604.html

Continuation of
http://mail.python.org/pipermail/python-dev/2002-November/030525.html

Guido got a hold of a OS X box and ran the test suite on it and found four
tests that didn't pass.  One was for ``locale`` (which Guido didn't care
about), another was the ``test_largefile`` test, another was for ``re``,
and the last one was for ``socket``.

The ``test_largefile`` suite seeks on a new file to a huge position.  On
OS X and Windows 2000 this actually adds to the file up to the seek point
so as to make it a complete file.  Windows 9x, on the other hand, just
writes to that offset regardless of whether it is within the bounds of the
file.  Because it creates such a huge file it takes forever to complete
and thus has been made an optional test on OS X.

The ``re`` tests were failing for a lack of space on the stack.  to allow
the tests to pass you need to execute the statement ``ulimit -s 2000``
which ups the stack size to approximately 2 Mb.  This has sense been added
to the regression testing framework to be done for you if needed.

``socket`` was failing on ``.getsockbyaddr()`` when trying it for the
machine's own name.  Barry Warsaw discovered that the test passed if you
set hostnames by using NetInfo.

==============================================================
`RE: how to kill process on Windows started with os.spawn?`__
==============================================================
__ http://mail.python.org/pipermail/python-dev/2002-December/030595.html

Skip Montanaro wanted to be able to kill a Windows process externally.  He
suggested implemented a ``os.kill()`` function that would kill a process
in a platform-independent way.  Problem is that Windows wants the process
hook which is different from what POSIX wants.  Another issue is that the
Windows docs say that you should not kill processes externally.  It seems
the idea has been dropped.

=========================================================
`extension module .o files wind up in the wrong place`__
=========================================================
__ http://mail.python.org/pipermail/python-dev/2002-December/030644.html

Skip Montanaro discovered that if you built Python in a subdirectory of
the source directory (e.g., source in ``source/`` and you are building in
``source/build/``) the .o files are put in the wrong place (they use
``source/`` as the root directory to place things instead of
``source/build/``).  Skip resolved his issue by doing a ``make distclean``
and then running ``../configure``.

=========================================
`Make universal newlines non-optional`__
=========================================
__ http://mail.python.org/pipermail/python-dev/2002-December/030692.html

Martin v. Lwis asked if anyone mind not making universal newline support a
compile option.  He volunteered to remove the ``#ifdef`` statements and
such to get this done.  Jack Jansen (who wrote the universal newline code)
pointed out that if it was required there could be issues on certain
platforms.  Martin had no issue with this since this would force the
platform maintainers to fix the code for it to work.  In the end Martin
suggested leaving it an option in 2.3 and removing it as an option and
requiring it in 2.4.

============================================
`Re: [Patches] Patch for xmlrpc encoding`__
============================================
__ http://mail.python.org/pipermail/python-dev/2002-December/030984.html

A discussion over a patch for ``xmlrpclib`` ended up on python-dev.  The
relevancy of it was that the statement ``sys.{get,set}defaultencoding`` is
only useful for coercion of Unicode to byte strings.  It was also stated
that ASCII should always be assumed to be the default encoding.  Also,
never put any non-ASCII text in a string; that is what Unicode objects are
for.

==========================
`PEP 242 Numeric kinds`__
==========================
__ http://mail.python.org/pipermail/python-dev/2002-December/030859.html

Paul DuBois emailed the list mentioning how a group of people were
supposed to get together to decide the fate of `PEP 242`_ in June 2002;
that didn't happen.  So Paul suggested making the suggested "making it a
separate deliverable on the Numeric_ site and withdrawing the PEP".
Everyone who responded pretty much said  to add the proposed module.  A
final outcome has not been decided as to whether Paul will rescind the PEP
or push it forward.  If you have an opinion on the matter, let it be
known.

.. _PEP 242: http://www.python.org/peps/pep-0242.html
.. _Numeric: http://www.pfdubois.com/numpy/

==================
`We got leaks!`__
==================
__ http://mail.python.org/pipermail/python-dev/2002-December/030933.html

Tim Peters discovered that the new datetime_ type was leaking memory.  The
C version was slowly leaking about 10 references per loop through the
testing suite.  But the Python version was leaking 37 reference per loop
through.  Tim did his reference checking using this chunk of code::

 def test_main():
    import gc
    r = unittest.TextTestRunner(stream=sys.stdout, verbosity=2)
    s = test_suite()
    while True:
        r.run(s)
        gc.collect()
        print gc.garbage  # this is always an empty list
        print '*' * 10, 'total refs:', sys.gettotalrefcount()

This substituted for the ``test_main()`` for the testing suite printed out
the reference numbers rather nicely.

Well, it didn't take the weekend to solve.  Kevin Jacobs stepped up to
help.  The first thing he found was that it was not occurring in a debug
build (which is why when you are testing new code you should **always**
run it in a debug build).  Then Kevin discovered the issue came up when
executing a ``__hash__()`` call on anything.  Tim then realized a missing
decrement in ``slot_tp_hash()`` was the culprit.  Then Kevin found another
culprit in ``cPickle_``.  That was also patched (and everything has been
flagged to be backported to the 2.2 branch).

The reason this thread is being mentioned is because it 1) was all solved
in 5 hours and 40 minutes which is pretty cool and 2) using the  chunk of
code that Tim provided to stare at the reference count is a good way to
look for memory leaks in your code.

.. _datetime:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/nondist/sandbox/datetime/
.. _cPickle: http://www.python.org/doc/current/lib/module-cPickle.html

=========================
`os.path.commonprefix`__
=========================
__ http://mail.python.org/pipermail/python-dev/2002-December/030947.html

Armin Rigo noticed how ``os.path.commonprefix()`` works
character-by-character and not by path separators as one might expect
(although the  documentation does warn about its char-by-char nature).
Skip Montanaro suggested ``os.path.commonpathprefix()`` be added.  He
mentioned that `Tools/scripts/trace.py`_ had such functionality.  As of
this writing no one has taken the initiative to add the function.

.. _Tools/scripts/trace.py:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Tools/scripts/trace.py

========================================================
`__getstate__() not inherited when __slots__ present`__
========================================================
__ http://mail.python.org/pipermail/python-dev/2002-December/030980.html

Greg Ward noticed that "If a class and its superclass both define
__slots__, it appears that __getstate__() is not inherited from the
superclass."  He wasn't sure if this was a bug or a feature.  Well, it's a
feature; since there  is no guarantee that the superclass's
``__getstate__()`` will know about the subclass's ``__slots__``, it is not
automatically inherited.

Kevin Jacobs provided the following snippet of code to deal with this
"feature"::

  def __getstate__(self):
    state = getattr(self, '__dict__', {}).copy()
    for obj in type(self).mro():
      for name in getattr(obj,'__slots__',()):
        if hasattr(self, name):
          state[name] = getattr(self, name)
    return state

  def __setstate__(self, state):
    for key,value in state.items():
      setattr(self, key, value)

====================
`Tarfile support`__
====================
__ http://mail.python.org/pipermail/python-dev/2002-December/031086.html

Gustavo Niemeyer asked for some help in reviewing the patch (
http://www.python.org/sf/651082 ) that adds Lars Gustabel's tarfile module
to the stdlib.  The question of the license came up.  Lars said he would
be willing to change the license over to the `Python Software Foundation`_
.  An initial license hand-off agreement is available at
http://www.python.org/psf/psf-contributor-agreement.html .

.. _Python Software Foundation: http://www.python.org/psf/

==============================
`Tcl, Tkinter, and threads`__
==============================
__ http://mail.python.org/pipermail/python-dev/2002-December/031107.html

Martin v. Lwis has now modified ```_tkinter`_`` as so to be able to be
used when Tcl has been compiled with thread support.  Special thanks goes
out to Martin for taking over maintenance of the module.

.. __tkinter:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/_tkinter.c

============
`PEP 298`__
============
__ http://mail.python.org/pipermail/python-dev/2002-December/031109.html

Splinter threads:

	- `PEP-298: buffer interface widening too much?
<http://mail.python.org/pipermail/python-dev/2002-December/031163.html>`__
	- `PEP-298: buffer interface widening too much?
<http://mail.python.org/pipermail/python-dev/2002-December/031184.html>`__

Thomas Heller wanted to resolve whether to move forward with an
implementation for `PEP 298`_ .  This led to a discussion over the
specifics of the PEP.  Martin v. Lwis wnated some more specific wording
from the PEP.

Todd Miller then stepped in to say he didn't like how much of an interface
was being tacked on.  Martin commented that his `PEP 286`_ would help in
this whole situation, which Todd ended up agreeing with.

.. _PEP 298: http://www.python.org/peps/pep-0298.html
.. _PEP 286: http://www.python.org/peps/pep-0286.html



From mwh@python.net  Mon Dec 16 11:20:54 2002
From: mwh@python.net (Michael Hudson)
Date: 16 Dec 2002 11:20:54 +0000
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/test test_descr.py,1.169,1.170
In-Reply-To: gvanrossum@users.sourceforge.net's message of "Fri, 13 Dec 2002 09:49:47 -0800"
References: <E18MtwZ-0001xK-00@sc8-pr-cvs1.sourceforge.net>
Message-ID: <2mfzsyciqx.fsf@starship.python.net>

gvanrossum@users.sourceforge.net writes:

> Update of /cvsroot/python/python/dist/src/Lib/test
> In directory sc8-pr-cvs1:/tmp/cvs-serv7366/Lib/test
> 
> Modified Files:
> 	test_descr.py 
> Log Message:
> Tighten the tests for assignment to __bases__: disallow empty tuple.

Oh, I knew about that, but didn't think it was worth bothering with.
Sorry.

Cheers,
M.

-- 
  That being done, all you have to do next is call free() slightly
  less often than malloc(). You may want to examine the Solaris
  system libraries for a particularly ambitious implementation of
  this technique.          -- Eric O'Dell, comp.lang.dylan (& x-posts)


From Jack.Jansen@cwi.nl  Mon Dec 16 13:15:07 2002
From: Jack.Jansen@cwi.nl (Jack Jansen)
Date: Mon, 16 Dec 2002 14:15:07 +0100
Subject: [Python-Dev] PEP 298
In-Reply-To: <200212131548.gBDFmrm09751@odiug.zope.com>
Message-ID: <659DE131-10F8-11D7-BB61-0030655234CE@cwi.nl>

On Friday, Dec 13, 2002, at 16:48 Europe/Amsterdam, Guido van Rossum 
wrote:

>> The PyArg_Parse replacement I would like to do (for which I have had,
>> unfortunately, absolutely no support up to now, and which I don't feel
>> up to designing all by myself) could fix this with PEP298: it would 
>> use
>> the locked buffer interface if available on the object, and release 
>> the
>> locked buffer after your method returns.
>
> I can't give you much more than moral support. :-(
>
> It sounds like it would be a Big Change though; are you sure you're up
> to this for Python 2.3?

No, I don't think it'll happen before that time, there's oodles of 
MacOSX stuff
I still want to do, and that'll keep me off the street for the time 
being.

But if we have the process underway before 2.3 I hope I may be able to 
convince
you that we should keep PyArg_Parse semantics as they are.
--
Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma 
Goldman



From guido@python.org  Mon Dec 16 13:30:41 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 08:30:41 -0500
Subject: [Python-Dev] PEP 298
In-Reply-To: Your message of "Mon, 16 Dec 2002 14:15:07 +0100."
 <659DE131-10F8-11D7-BB61-0030655234CE@cwi.nl>
References: <659DE131-10F8-11D7-BB61-0030655234CE@cwi.nl>
Message-ID: <200212161330.gBGDUfi06503@pcp02138704pcs.reston01.va.comcast.net>

> But if we have the process underway before 2.3 I hope I may be able
> to convince you that we should keep PyArg_Parse semantics as they
> are.

Twist my arm! :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jmiller@stsci.edu  Mon Dec 16 14:28:28 2002
From: jmiller@stsci.edu (Todd Miller)
Date: Mon, 16 Dec 2002 09:28:28 -0500
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira>	<3DFCFD21.2090707@stsci.edu> <m3fzsylttd.fsf@mira.informatik.hu-berlin.de>
Message-ID: <3DFDE30C.7080009@stsci.edu>

Martin v. Löwis wrote:

>Todd Miller <jmiller@stsci.edu> writes:
>
>  
>
>>I think that kind of backwards compatability just paves the wave for
>>extensions to stay broken and complicates the object structures as
>>well. I guess PEP-298 does create the possibility for creating correct
>>extensions,  I just hate the thought of 5 getpointer calls where there
>>are now 3 and should be 2.
>>    
>>
>
>I think this is the core of your objection. However, notice that all
>
You're right I think,  but I also percieve the complexity of extension 
writing going up as well.   Perhaps it's a false perception which can be 
dispelled by documentation stating that new extensions should *always* 
use the locking interface,  with the old interface preserved for the 
sole purpose of backwards compatability.  Without some thought,  the 
following was not clear:  the need for locking is a property of specific 
types,  not specific extension functions.  Therefore the existence of 
two get pointer varieties implies an ambguity which is not really there: 
 should I use the locking or non-locking variety?  I think the answer is 
always:  use the locking variety.

>types implementing have to be modified in either proposal, and that
>adding the two getpointer functions should be trivial in most cases:
>- if the type has read/write memory, implement them both the same
>  way;
>- if the type has read-only memory, set the write pointer to NULL
>- make the getpointer function a wrapper around the existing
>  getpointer, requesting only a single segment and simultaneously
>  locking the buffer.
>
>The overhead implementation-wise is really minimal; all the hard work
>will come from the locking code - see Thomas' patch to the string
>  
>
Thanks for the pointer.

>object for an example. We could even provide a Generic function which
>delegates the new get functions to the existing ones, accessing
>segment zero.
>  
>
Wouldn't this require more information about the lock implementation? 
 As PEP-298 stands,  I thought that was left up to the buffered type. 
 It might make sense to change the locking getpointer fields into lock, 
and release into unlock.  Then compose the primitive operations in the 
wrapper as you suggested.   That wouldn't fix the size_t problem though.

>  
>
>>Sounds good to me.  I hope you implement it!  If PEP-286 were here
>>now, a simplified PEP-298 would be possible. 
>>    
>>
>
>I now recall what the issue is: you can't assume that modifying the
>argument tuple object to record resources is feasible, as this breaks
>apply (where in every case the user allocates a tuple, and where this
>is a plain old tuple).
>  
>
Is the efficiency of apply() so critical that a conversion is ruled out?

>So I'm now envisioning a strategy where a thread-local
>(i.e. thread-state associated) stack is maintained - on argument tuple
>allocation, a frame is put on this stack, and all resources go into
>that frame, on tuple decrefing, the frame is cleared. There would be
>three kinds of resources: objects, void*, and locked buffers. For
>efficiency, this stack and its frames would be a flat memory block.
>  
>
Way over my head here...  I guess the point is that arg tuples aren't 
the only point of attack.

>In any case, I'm still -0 on PEP 298 as it stands: it should deal with
>much more aspects of buffers (and I now consider argument parsing to
>be one of them), when those things are added, I'll support the PEP -
>your objection that this adds too much API is of no concern to me.
>  
>
Well,  thanks for discussing it anyway.   I had a feeling this 
conversation would end this way before I ever opened my mouth,  but I 
wanted to change the balance between saying something and saying *nothing*.

>Regards,
>Martin
>
Regards,
Todd

>  
>



From pje@telecommunity.com  Mon Dec 16 15:15:01 2002
From: pje@telecommunity.com (Phillip J. Eby)
Date: Mon, 16 Dec 2002 10:15:01 -0500
Subject: [Python-Dev] Zip import and sys.path manipulation (was Re: directory
 hierarchy proposal)
In-Reply-To: <3DFDCE88.1020709@zope.com>
References: <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com>
 <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com>
 <20021215211435.V11257@prim.han.de>
Message-ID: <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>

(CC'd to python-dev because of possible impact on zipimport implementation/API)

At 08:00 AM 12/16/02 -0500, Jim Fulton wrote:
>Each zope package __init__ file would have a bit of boilerplate code
>that would include all zope directories in sys.path:
>
>   # Boilerplate code that combines all zope directories in sys.path
>   import sys, os
>   __path__ = [os.path.join(p, 'zope') for p in sys.path]
>
>Jim

That should probably read:

__path__ = [os.path.join(p, 'zope') for p in sys.path if 
isinstance(p,StringTypes)]

or else it could break in 2.3 if somebody's putting non-strings on 
sys.path.  I don't remember exactly how the python-dev discussion turned 
out, but I kind of got the impression that it was going to be allowable in 
2.3.  Also, I'm pretty sure it will not give the expected/desired results 
in conjunction with zipfile imports.

For these and related reasons, I would suggest that a slightly different 
snippet be used instead (the names are lousy, improvements welcome):

from namespace_packages import makeNamespacePath
__path__ = makeNamespacePath('zope')

And then make whatever carries "namespace_packages" be something that's 
distributed separately from Zope, preferably part of the standard library 
or distutils (and backported where possible).       If we can make it so 
that the __init__ code never *has* to change, that will minimize the 
likelihood of incompatible __init__.py's floating around.  Meanwhile, a 
module that actually *knows* something about what can be on sys.path (and 
how to interpret it) can do so.

In order to support something like this, importer objects would need to be 
able to return something that's the moral equivalent of "os.path.join" on 
an existing sys.path entry; either another importer object or a string that 
can be handled by one.  At this point, checking for a '.zip' extension at 
the end of a string isn't going to cut it any more.



From guido@python.org  Mon Dec 16 15:47:14 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 10:47:14 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 10:15:01 EST."
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
Message-ID: <200212161547.gBGFlEB01302@odiug.zope.com>

> (CC'd to python-dev because of possible impact on zipimport
> implementation/API)
> 
> At 08:00 AM 12/16/02 -0500, Jim Fulton wrote:
> >Each zope package __init__ file would have a bit of boilerplate code
> >that would include all zope directories in sys.path:
> >
> >   # Boilerplate code that combines all zope directories in sys.path
> >   import sys, os
> >   __path__ = [os.path.join(p, 'zope') for p in sys.path]
> >
> >Jim
> 
> That should probably read:
> 
> __path__ = [os.path.join(p, 'zope') for p in sys.path if 
> isinstance(p,StringTypes)]
> 
> or else it could break in 2.3 if somebody's putting non-strings on
> sys.path.  I don't remember exactly how the python-dev discussion
> turned out, but I kind of got the impression that it was going to be
> allowable in 2.3.

Correct.  While the standard setup will still have only strings, we'll
support non-strings on sys.path and some users may want to use this.

> Also, I'm pretty sure it will not give the expected/desired results
> in conjunction with zipfile imports.

I think it will, actually; if /path/foo.zip is a zip file on sys.path,
/path/foo.zip/blah references a directory inside that zip file.

> For these and related reasons, I would suggest that a slightly
> different snippet be used instead (the names are lousy, improvements
> welcome):
> 
> from namespace_packages import makeNamespacePath
> __path__ = makeNamespacePath('zope')
> 
> And then make whatever carries "namespace_packages" be something
> that's distributed separately from Zope, preferably part of the
> standard library or distutils (and backported where possible).  If
> we can make it so that the __init__ code never *has* to change, that
> will minimize the likelihood of incompatible __init__.py's floating
> around.  Meanwhile, a module that actually *knows* something about
> what can be on sys.path (and how to interpret it) can do so.

This is not a bad idea: put the logic to add stuff to a package's
__path__ according to a certain policy in the standard library.

(Of course, getting it into Python 2.2 would require another release,
2.2.3 -- which may happen anyway.)

> In order to support something like this, importer objects would need
> to be able to return something that's the moral equivalent of
> "os.path.join" on an existing sys.path entry; either another
> importer object or a string that can be handled by one.  At this
> point, checking for a '.zip' extension at the end of a string isn't
> going to cut it any more.

As I said above, I think it will.

Time's running short (we're close to releasing Python 2.3 alpha 1);
we should decide on a name and a specific policy ASAP.  I propose the
following:

  from pkgutil import extended_path
  __path__ = extended_path(__path__, __name__)

Where pkgutil.py could be something like this:

  import sys.path

  def extended_path(path, name):
      """Extend a package's path.  (XXX more.)"""
      path = path[:]
      for dir in sys.path:
          if isinstance(dir, (str, unicode)):
              dir = os.path.join(dir, name)
              if dir not in path and os.path.isdir(dir):
                  path.append(dir)
      return path

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Mon Dec 16 16:02:47 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 16 Dec 2002 11:02:47 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory
 hierarchy proposal)
References: <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com>
 <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com>
 <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
Message-ID: <15869.63783.430525.295170@gargle.gargle.HOWL>

>>>>> "PJE" == Phillip J Eby <pje@telecommunity.com> writes:

    PJE> That should probably read:

    | __path__ = [os.path.join(p, 'zope') for p in sys.path if 
    | isinstance(p,StringTypes)]

    PJE> or else it could break in 2.3 if somebody's putting
    PJE> non-strings on sys.path.  I don't remember exactly how the
    PJE> python-dev discussion turned out, but I kind of got the
    PJE> impression that it was going to be allowable in 2.3.  Also,
    PJE> I'm pretty sure it will not give the expected/desired results
    PJE> in conjunction with zipfile imports.

    PJE> For these and related reasons, I would suggest that a
    PJE> slightly different snippet be used instead (the names are
    PJE> lousy, improvements welcome):

    | from namespace_packages import makeNamespacePath
    | __path__ = makeNamespacePath('zope')

Random, harried thought: maybe we can use .pth files to extend the
__path__?  IOW, something like the above would search sys.path for
zope.pth files, and do the equivalent to what site.py already does.

-Barry


From guido@python.org  Mon Dec 16 16:07:27 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 11:07:27 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 11:02:47 EST."
 <15869.63783.430525.295170@gargle.gargle.HOWL>
References: <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <15869.63783.430525.295170@gargle.gargle.HOWL>
Message-ID: <200212161607.gBGG7Rs01448@odiug.zope.com>

> Random, harried thought: maybe we can use .pth files to extend the
> __path__?  IOW, something like the above would search sys.path for
> zope.pth files, and do the equivalent to what site.py already does.

That would be a useful addition to the pkgutil.py module that I just
proposed.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Paul.Moore@atosorigin.com  Mon Dec 16 16:12:40 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Mon, 16 Dec 2002 16:12:40 -0000
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com>

Guido said:

> Time's running short (we're close to releasing Python 2.3 alpha 1);
> we should decide on a name and a specific policy ASAP.  I propose the
> following:
>=20
>   from pkgutil import extended_path
>   __path__ =3D extended_path(__path__, __name__)
>=20
> Where pkgutil.py could be something like this:
>=20
>   import sys.path
>=20
>   def extended_path(path, name):
>       """Extend a package's path.  (XXX more.)"""
>       path =3D path[:]
>       for dir in sys.path:
>           if isinstance(dir, (str, unicode)):
>               dir =3D os.path.join(dir, name)
>               if dir not in path and os.path.isdir(dir):
>                   path.append(dir)
>       return path

Are you proposing this for the standard library? It certainly
sounds reasonable, but the documentation should make clear
that it still fails for certain types of sys.path entry.

Specifically,

  - string subclasses will get converted into basic strings
  - strings which don't have a path structure wil break

(and of course, objects other than string subclasses don't get
"extended").

I don't think that either of these is even remotely a
showstopper, but they do need to be documented - specifically
because as a module, this code can be imported by anybody, and so
the constraints on the user should be made clear.

In the long term, I think the two most likely things to appear on
sys.path, apart from directory names, are

  - "Filesystem namespace extensions" like the zipfile support.
    That is, things which act like virtual directories.
  - raw importer objects

Frankly, just dumping a raw imported onto sys.path (or a package
path) is so much easier than implementing a string naming
scheme, that unless there's an overwhelmingly natural string
representation (virtual directory or URL are the only two that
come to mind) it's simpler not to bother.

So saying that module paths should only contain pathlike strings,
or importer objects, is probably reasonable. I wouldn't recommend
enforcing this, just documenting it as the normal convention.
Modules can document that they don't work with applications that
violate this convention, and relentless experimentalists still
have the freedom to break the recommendation (at their own risk!)
if they want to see what happens...

Paul.


From barry@zope.com  Mon Dec 16 16:16:19 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 16 Dec 2002 11:16:19 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
References: <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com>
 <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com>
 <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <15869.63783.430525.295170@gargle.gargle.HOWL>
 <200212161607.gBGG7Rs01448@odiug.zope.com>
Message-ID: <15869.64595.601832.869032@gargle.gargle.HOWL>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    >> Random, harried thought: maybe we can use .pth files to extend
    >> the __path__?  IOW, something like the above would search
    >> sys.path for zope.pth files, and do the equivalent to what
    >> site.py already does.

    GvR> That would be a useful addition to the pkgutil.py module that
    GvR> I just proposed.

The nice thing is if we can localize things to pkgutil.py, we wouldn't
/have/ to release Python 2.2.3 to get backported support.  Just
release pkgutil.py as its own distutils package which Zope3 could
"setup.py install" first thing.  It lands in site-packages for older
Python's to use, but when it gets released with Python 2.3, it'll be
ignored in favor of the standard library version.

-Barry


From guido@python.org  Mon Dec 16 16:22:13 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 11:22:13 -0500
Subject: [Python-Dev] [Python-checkins] python/dist/src/Objects abstract.c,2.93.6.7,2.93.6.8
In-Reply-To: Your message of "Fri, 13 Dec 2002 20:26:35 EST."
 <BA1FF2F9.2D123%goodger@python.org>
References: <BA1FF2F9.2D123%goodger@python.org>
Message-ID: <200212161622.gBGGMD401543@odiug.zope.com>

> This backport checkin (Oct 5 2002) changed the exception text between
> 2.2.1:
> 
>     >>> int(None)
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>     TypeError: object can't be converted to int
> 
> and 2.2.2 (last line):
> 
>     >>> int(None)
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>     TypeError: int() argument must be a string or a number
> 
> Should this change have been backported to 2.2.2 from the 2.3
> codebase?

Clarifications of error messages is acceptable for bugfix releases.
Some error messages need more clarification than others, and you can
argue about this particular one, but I don't think you can rely on all
error messages remaining the same within bugfix releases.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec 16 16:24:01 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 11:24:01 -0500
Subject: [Python-Dev] Deprecate doctest?
In-Reply-To: Your message of "Sat, 14 Dec 2002 14:42:15 EST."
 <20021214194215.GA9536@panix.com>
References: <BA1FF2F9.2D123%goodger@python.org> <3DFB101D.6010306@lemburg.com>
 <20021214194215.GA9536@panix.com>
Message-ID: <200212161624.gBGGO2j01562@odiug.zope.com>

> That's tricky, given that doctest is in the standard library, and that's
> exactly what doctest does.  Are you advocating deprecating doctest?

You gotta be kidding.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pje@telecommunity.com  Mon Dec 16 16:23:47 2002
From: pje@telecommunity.com (Phillip J. Eby)
Date: Mon, 16 Dec 2002 11:23:47 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re:
 directory hierarchy proposal)
In-Reply-To: <200212161547.gBGFlEB01302@odiug.zope.com>
References: <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com>
 <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com>
 <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
Message-ID: <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>

(Summary: good plan for name and place, issues exist w/sketched implementation)

At 10:47 AM 12/16/02 -0500, Guido van Rossum wrote:

>Time's running short (we're close to releasing Python 2.3 alpha 1);
>we should decide on a name and a specific policy ASAP.  I propose the
>following:
>
>   from pkgutil import extended_path
>   __path__ = extended_path(__path__, __name__)

How would that work for say, 'zope.app'?


>Where pkgutil.py could be something like this:
>
>   import sys.path
>
>   def extended_path(path, name):
>       """Extend a package's path.  (XXX more.)"""
>       path = path[:]
>       for dir in sys.path:
>           if isinstance(dir, (str, unicode)):
>               dir = os.path.join(dir, name)
>               if dir not in path and os.path.isdir(dir):
>                   path.append(dir)
>       return path
>
>--Guido van Rossum (home page: http://www.python.org/~guido/)

The 'isdir()' part isn't going to work on zipfiles...  unless os.path is 
going to have zipfile support in 2.3.  :)



From guido@python.org  Mon Dec 16 16:27:52 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 11:27:52 -0500
Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python
In-Reply-To: Your message of "Sat, 14 Dec 2002 21:06:17 EST."
 <200212142106.01812.mclay@nist.gov>
References: <200212142106.01812.mclay@nist.gov>
Message-ID: <200212161627.gBGGRq801621@odiug.zope.com>

> sf/653938I is a patch to integrate the fixedpoint module, that was
> created by Tim Peters, into Python as a first class object. That is,
> the patch adds this new number type so that it has direct syntax
> support like float, int, long, str., etc. within the interpreter.

(a) I don't want any (more) change to the syntax for Python 2.3.

(b) Like the sets module, proposals for new datatypes should first be
    prototyped as standard library modules before being promoted to
    built-in status (with their own literals).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec 16 16:31:15 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 11:31:15 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: Your message of "Sat, 14 Dec 2002 16:29:27 +0100."
 <015b01c2a385$97010700$6d94fea9@newmexico>
References: <015b01c2a385$97010700$6d94fea9@newmexico>
Message-ID: <200212161631.gBGGVFB01638@odiug.zope.com>

> latest CVS
> 
> Python 2.3a0 (#1, Dec 14 2002, 07:19:32)
> [GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> 
> consider
> 
> >>> import rexec
> >>> r=rexec.RExec()
> >>> r.r_exec("class C: pass\nC.a=0")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/home/users/p/pe/pedronis/python/dist/src/Lib/rexec.py", line 309, in
> r_
> exec
>     exec code in m.__dict__
>   File "<string>", line 2, in ?
> RuntimeError: classes are read-only in restricted mode
> 
> vs.
> 
> >>> r.r_exec("class C(object): pass\nC.a=0")
> >>>

Hm...  Do you see any way to break out of restricted execution mode
using this?  I suppose a fix would be simple enough, but I'm more and
more inclined to simply rip out rexec from the distribution -- it's
never going to be safe, and I doubt it's very useful as long as it's
not safe.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From barry@zope.com  Mon Dec 16 16:38:13 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 16 Dec 2002 11:38:13 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
References: <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com>
 <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com>
 <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <15869.63783.430525.295170@gargle.gargle.HOWL>
 <200212161607.gBGG7Rs01448@odiug.zope.com>
Message-ID: <15870.373.513177.215927@gargle.gargle.HOWL>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    >> Random, harried thought: maybe we can use .pth files to extend
    >> the __path__?  IOW, something like the above would search
    >> sys.path for zope.pth files, and do the equivalent to what
    >> site.py already does.

    GvR> That would be a useful addition to the pkgutil.py module that
    GvR> I just proposed.

BTW, in the spirit of TOOWTDI, here's a very quick hack...

-Barry

-------------------- snip snip --------------------
# Use like this:
#
# from pkgutil import extendpath
# __path__ = extendpath(__path__, __name__)

import os
import sys
from site import makepath

def _read(basedir, fp):
    dirs = {}
    # Stolen largely from site.py
    for line in fp:
        if line.startswith('#'):
            continue
        if line.startswith('import'):
            exec line
            continue
        if line.endswith('\n'):
            line = line[:-1]
        dir, dircase = makepath(basedir, line)
        if os.path.exists(dir):
            dirs[dir] = True
    return dirs


def extendpath(path, name):
    path = dict(zip(sys.path, [True for x in sys.path]))
    for dir in sys.path:
        if isinstance(dir, (str, unicode)):
            pth = os.path.join(dir, name + os.extsep + 'pth')
            try:
                fp = open(pth)
            except IOError:
                continue
            try:
                path.update(_read(dir, fp))
            finally:
                fp.close()
    return path.keys()


From guido@python.org  Mon Dec 16 16:42:34 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 11:42:34 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 16:12:40 GMT."
 <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com>
Message-ID: <200212161642.gBGGgYJ03474@odiug.zope.com>

> > Time's running short (we're close to releasing Python 2.3 alpha 1);
> > we should decide on a name and a specific policy ASAP.  I propose the
> > following:
> > 
> >   from pkgutil import extended_path
> >   __path__ = extended_path(__path__, __name__)
> > 
> > Where pkgutil.py could be something like this:
> > 
> >   import sys.path
> > 
> >   def extended_path(path, name):

(BTW I now think extend_path() is a better and simpler name for this.)

> >       """Extend a package's path.  (XXX more.)"""
> >       path = path[:]
> >       for dir in sys.path:
> >           if isinstance(dir, (str, unicode)):
> >               dir = os.path.join(dir, name)
> >               if dir not in path and os.path.isdir(dir):
> >                   path.append(dir)
> >       return path
> 
> Are you proposing this for the standard library?

Yes.

> It certainly sounds reasonable, but the documentation should make
> clear that it still fails for certain types of sys.path entry.
> 
> Specifically,
> 
>   - string subclasses will get converted into basic strings

I find string subclasses an ugly hack that should not be used.  The
meta hook (or whatever it's called now) should do this.

>   - strings which don't have a path structure wil break

We can fix that by skipping anything for which os.path.isdir(dir) [for
the original value of dir] isn't true.

> (and of course, objects other than string subclasses don't get
> "extended").

We can use some kind of extension hook to deal with those.  The
important point is that the idiom placed in __init__.py can remain the
same (the two lines quoted above) while the implementation of
pkgutil.extend_path() can evolve or be modified (even monkey-patched
if you're desperate) as the import hooks evolve.

The main reason for putting this in the stdlib is to avoid the need to
avoid the boilerplate code in everybody's __init__.py.

> I don't think that either of these is even remotely a
> showstopper, but they do need to be documented - specifically
> because as a module, this code can be imported by anybody, and so
> the constraints on the user should be made clear.

Of course.  Though with the current state of documentation, I give
less priority to documenting every little detail of this code than to
getting it in the code base (with *some* documentation, for sure).

> In the long term, I think the two most likely things to appear on
> sys.path, apart from directory names, are
> 
>   - "Filesystem namespace extensions" like the zipfile support.
>     That is, things which act like virtual directories.
>   - raw importer objects
> 
> Frankly, just dumping a raw imported onto sys.path (or a package
> path) is so much easier than implementing a string naming
> scheme, that unless there's an overwhelmingly natural string
> representation (virtual directory or URL are the only two that
> come to mind) it's simpler not to bother.

OTOH I expect that most things you'd want to place on sys.path *do*
have a natural string representation.  After all, if something doesn't
have a name, it's hard to talk about.  And if it has a name, it can be
placed in PYTHONPATH (with the minor quibble that URLs use ':' which
-- at least on Unix -- is also used as the separator in *PATH
variables. :-( )

> So saying that module paths should only contain pathlike strings,
> or importer objects, is probably reasonable. I wouldn't recommend
> enforcing this, just documenting it as the normal convention.
> Modules can document that they don't work with applications that
> violate this convention, and relentless experimentalists still
> have the freedom to break the recommendation (at their own risk!)
> if they want to see what happens...

Maybe importer objects should have a standard API that lets you ask
for an extended object.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mclay@nist.gov  Mon Dec 16 16:45:34 2002
From: mclay@nist.gov (Michael McLay)
Date: Mon, 16 Dec 2002 11:45:34 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEMIDCAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCEMIDCAB.tim.one@comcast.net>
Message-ID: <200212161145.21295.mmclay@comcast.net>

On Monday 16 December 2002 01:48 am, Tim Peters wrote:
> [Michael McLay]
>
> > ...
> > The semantics of the precision parameter may be incorrect because
> > it is only addressing the precision of the fraction part of the
> > number.
>
> That's why the module was called FixedPoint.

Fixing the decimal place would make sense for applications having to do with 
money. That is quite different than how numbers are treated in scientific 
calculations. That is were I borrowed the idea of significant digits. 
Accountants tend to think that all digits are significant, down to the last 
penny.

> > Shouldn't it reflect the number of significant digits?
>
> That would be more appropriate to a floating point scheme.  Aahz has been
> working on a Python implementation of Mike Cowlishaw's hybrid proposal:
>
>     http://www2.hursley.ibm.com/decimal/
>
> for about, oh, fifteen years now <wink>.

Wow, so Aahz anticipated the creation of Python and started early. This IS a 
dedicated community.

> > ...
> >
> > >>> 3.03e5d
> >
> > 303000.00d
> >
> > The representation is incorrect. It should have returned 3.03e5d
> > but it padded the result with bogus zeros.
>
> It seems you don't really want a fixed-point type.  The FixedPoint
> constructor avoided this rat hole by making the user be explicit about the
> fixed number of fractional digits (or accept the default 2).

My goal is to have a new numerical type added to Python that satisfies the 
requirements for accountants and newbies. Your description of fixed point 
seems to meet this goal. My confusion was raised by the 'e' notation seeming 
to add precision to numbers (e.g. 1.0e5 expands to 100000.0.) This would be 
incorrect for reporting significant digits in scientific papers, but the 
rules for accounting math are different. This was simply my confusion. 
FixedPoint appears to be sufficient for bankers and accountants.

---snip Tim's explanation of roundoff error, thanks for the clarification--- 

> That's really representation error.  Until the distinction becomes clear,
> you'll keep believing that decimal arithmetic is somehow better <wink>.

I get the distinction now, thanks. 

---snip discussion of rationales and division errors---
> [Brett Cannon]
>
> > ...
> > It is still going to take someone to champion the module to get BDFL
> > pronouncment from Guido autonomously or get enough people to rally
> > behind to put pressure on Guido to let it in.
>
> Guido already agreed to fold FixedPoint into 2.3, provided someone other
> than me do the work <wink>.  Doug Fort has done a great job on it
> (including LaTeX docs, Fred!), and we're just missing someone with the
> time/desire to do final integration.
>
> That's independent of Michael's patch, of course.  The FixedPoint module is
> just another library module.

-- snip Tim's comments on when his stuff gets added to Python --

> If Python were to grow a native decimal type, I'd rather move to
> Cowlishaw's design, because it has the best shot at becoming a standard
> influential enough to sway HW design (decimal fp in HW wouldn't be notably
> more expensive or slower than binary fp in HW, given current chip
> capacities; this was a much bigger deal when chip real estate was scarce).

This is a very good long term goal. So for Python3k the standard numerical 
type will be Cowlishaw's decimal numbers. To bad the decNumber library from 
IBM is not open source. 

In the short term what do you advise be done with the patch to add a fixed 
point literal and a builtin fixed point conversion function? I could change 
the name from decimal() to FixedPoint(), but the capital letters in the name 
scream "ugly". This isn't consistent with the short names used by the other 
numerical conversion functions (int, float, hex, oct...). Would fixed() be an 
acceptable name? This also suggests the suffix should be changed to 'f' and 
'F' instead of 'd' and 'D'. 

> >> The syntax change is primarily to make Python more appealing to
> >> professions that require the semantics of fixed point decimal numbers.
> >
> > I understand that, but who knows how many of those types of users are
> > out there (current and potentiol).
>
> "Do applications actually use decimal data?" from Cowlishaw's FAQ should go
> a way toward answering that:
>
>     http://www2.hursley.ibm.com/decimal/decifaq.html

According to this reference the most frequent numerical column type used in 
commercial databases is decimal (55%). The float type is used just 1.4% of 
the time with the rest being split between Integer and SmallInt. The data was 
taken from an analysis of applications used by Airline systems, Banking, 
Financial Analysis, Insurance, Inventory control, Management reporting, 
Marketing services, Order entry, Order processing, Pharmaceutical 
applications, and Retail sales. Is it safe to conclude that Python will be be 
much more appealing to a huge segment of the IT market if grows a native 
decimal number type?



From barry@python.org  Mon Dec 16 16:47:39 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 16 Dec 2002 11:47:39 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com>
 <200212161642.gBGGgYJ03474@odiug.zope.com>
Message-ID: <15870.939.16954.763349@gargle.gargle.HOWL>


    GvR> (BTW I now think extend_path() is a better and simpler name
    GvR> for this.)

Or even extendpath() <wink>>
-Barry


From guido@python.org  Mon Dec 16 16:54:27 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 11:54:27 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 11:23:47 EST."
 <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
References: <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
Message-ID: <200212161654.gBGGsRe03537@odiug.zope.com>

> At 10:47 AM 12/16/02 -0500, Guido van Rossum wrote:
> 
> >Time's running short (we're close to releasing Python 2.3 alpha 1);
> >we should decide on a name and a specific policy ASAP.  I propose the
> >following:
> >
> >   from pkgutil import extended_path
> >   __path__ = extended_path(__path__, __name__)
> 
> How would that work for say, 'zope.app'?

Good point.  Back to the drawing board; I'll have to turn dots in the
name into os.sep.

> >Where pkgutil.py could be something like this:
> >
> >   import sys.path
> >
> >   def extended_path(path, name):
> >       """Extend a package's path.  (XXX more.)"""
> >       path = path[:]
> >       for dir in sys.path:
> >           if isinstance(dir, (str, unicode)):
> >               dir = os.path.join(dir, name)
> >               if dir not in path and os.path.isdir(dir):
> >                   path.append(dir)
> >       return path
> 
> The 'isdir()' part isn't going to work on zipfiles...  unless os.path is 
> going to have zipfile support in 2.3.  :)

I tried putting Zope3 in a zipfile, and it's hopeless; lots and lots
of code expects to find data files (at least page templates and zcml
files; probably also style sheets and other stuff that I don't even
know about) by taking a module's __file__, stripping the filename to
get a directory, and concatenating the directory with the name of a
known data file.  Plus there's the issue of extension modules.

So I think the isdir() not working for zip files can be considered a
feature. :-)

BTW, here's a more recent draft, incorporating a search for .pth
files and with a different place for the isdir():

## pkgutil.py #################################################################
"""Utilities to support packages."""

import os
import sys

def extend_path(path, name):
    """Extend a package's path.

    Intended use is to place the following code in a package's __init__.py:

        from pkgutil import extend_path
        __path__ = extend_path(__path__, __name__)

    This will add to the package's __path__ all subdirectories of
    directories on sys.path named after the package.  This is useful
    if one wants to distribute different parts of a single logical
    package as multiple directories.
    """
    path = path[:]
    for dir in sys.path:
        if isinstance(dir, (str, unicode)) and os.path.isdir(dir):
            subdir = os.path.join(dir, name)
            if subdir not in path and os.path.isdir(subdir):
                path.append(subdir)
            pthfile = os.path.join(dir, name + os.extsep + "pth")
            if os.path.isfile(pthfile):
                try:
                    f = open(pthfile)
                except IOError, msg:
                    sys.stderr.write("Can't open %s: %s\n" %
                                     (pthfile, msg))
                else:
                    for line in f:
                        line = line.strip()
                        if not line or line.startswith('#'):
                            continue
                        path.append(line) # Don't check for existence!
                    f.close()
    return path
###############################################################################

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mwh@python.net  Mon Dec 16 16:57:36 2002
From: mwh@python.net (Michael Hudson)
Date: 16 Dec 2002 16:57:36 +0000
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: Michael McLay's message of "Mon, 16 Dec 2002 11:45:34 -0500"
References: <LNBBLJKPBEHFEDALKOLCCEMIDCAB.tim.one@comcast.net> <200212161145.21295.mmclay@comcast.net>
Message-ID: <2m8yypdhq7.fsf@starship.python.net>

Michael McLay <mclay@nist.gov> writes:

> On Monday 16 December 2002 01:48 am, Tim Peters wrote:
> > [Michael McLay]
> >
> > > ...
> > > The semantics of the precision parameter may be incorrect because
> > > it is only addressing the precision of the fraction part of the
> > > number.
> >
> > That's why the module was called FixedPoint.
> 
> Fixing the decimal place would make sense for applications having to do with 
> money. That is quite different than how numbers are treated in scientific 
> calculations. That is were I borrowed the idea of significant digits. 
> Accountants tend to think that all digits are significant, down to the last 
> penny.

I'm a little confused; you're borrowing terms from scientific papers
to explain things to accountants?

[...]
> This is a very good long term goal. So for Python3k the standard numerical 
> type will be Cowlishaw's decimal numbers.

*I* would hope not; there are reasons to prefer binary fp to decimal
fp if you're not a money man.

Cheers,
M.

-- 
  ... with these conditions cam the realisation that ... nothing
  turned a perfectly normal healthy individual into a great political
  or military leader better than irreversible brain damage.
                   -- The Hitch-Hikers Guide to the Galaxy, Episode 11


From guido@python.org  Mon Dec 16 17:00:02 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 12:00:02 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 11:47:39 EST."
 <15870.939.16954.763349@gargle.gargle.HOWL>
References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> <200212161642.gBGGgYJ03474@odiug.zope.com>
 <15870.939.16954.763349@gargle.gargle.HOWL>
Message-ID: <200212161700.gBGH02S03607@odiug.zope.com>

>     GvR> (BTW I now think extend_path() is a better and simpler name
>     GvR> for this.)
> 
> Or even extendpath() <wink>>

Jim doesn't like sequences of lowercase words strung together without
using either camelcase or underscores.  In this particular case I
agree.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pje@telecommunity.com  Mon Dec 16 17:02:28 2002
From: pje@telecommunity.com (Phillip J. Eby)
Date: Mon, 16 Dec 2002 12:02:28 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re:
 directory hierarchy proposal)
In-Reply-To: <200212161654.gBGGsRe03537@odiug.zope.com>
References: <Your message of "Mon, 16 Dec 2002 11:23:47 EST." <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
 <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com>
 <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com>
 <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
Message-ID: <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com>

At 11:54 AM 12/16/02 -0500, Guido van Rossum wrote:

>I tried putting Zope3 in a zipfile, and it's hopeless; lots and lots
>of code expects to find data files (at least page templates and zcml
>files; probably also style sheets and other stuff that I don't even
>know about) by taking a module's __file__, stripping the filename to
>get a directory, and concatenating the directory with the name of a
>known data file.  Plus there's the issue of extension modules.
>
>So I think the isdir() not working for zip files can be considered a
>feature. :-)

Alternately, now that there's a "pkgutils" to put it in, why not add a 
"fileNearModule()" function that returns a file handle opened for reading 
on the basis of a module name and a file name?  I currently use something 
roughly like:

def fileNearModule(moduleName,filename):
     basedir = os.path.dirname(sys.modules[moduleName].__file__)
     return open(os.path.join(basedir,filename), 'r')

Of course, a zipfile version would need some work, and clearly the zipfile 
version wouldn't be a candidate for alpha 1.

Anyway, if we could establish an API for some of these sorts of things in 
pkgutils, then they could be the official correct way to get at data files 
and make zipfiles work for a lot more things long term.



From Paul.Moore@atosorigin.com  Mon Dec 16 17:05:28 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Mon, 16 Dec 2002 17:05:28 -0000
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619988@UKDCX001.uk.int.atosorigin.com>

From: Guido van Rossum [mailto:guido@python.org]

> I find string subclasses an ugly hack that should not be used.

Agreed.

> The meta hook (or whatever it's called now) should do this.

Why bother?

> >   - strings which don't have a path structure wil break
>
> We can fix that by skipping anything for which
> os.path.isdir(dir) [for the original value of dir] isn't true.

Agreed. This makes extend_path "do the right thing" for anything
that can currently be on a path, and punt to an extension hook
for anything else. That sounds reasonable (but see below).

> > (and of course, objects other than string subclasses don't
> > get "extended").
>
> We can use some kind of extension hook to deal with those.

It depends where you envisage that hook. I'd be unhappy with
requiring something in the import hook implementation - two
reasons, first because the current hook mechanism is nice and
simple, and hence easy to document and use, and second because
people simply won't implement the path-extension hook, and nearly
everything will work fine - the breakage only occurs when the
hook is used in conjunction with pkgutil. At that point, saying
"I told you so" isn't much help...

I've seen this sort of thing happen too often before to be
comfortable with it. A simple protocol, with an obscure little
requirement which is to support a relatively rare usage. People
*never* implement that correctly, and the result is annoying
breakages.

> The important point is that the idiom placed in __init__.py
> can remain the same (the two lines quoted above) while the
> implementation of pkgutil.extend_path() can evolve or be
> modified (even monkey-patched if you're desperate) as the
> import hooks evolve.

I'd say the extension mechanism should be external to both
the import hook and pkgutil. It then becomes the user's
responsibility, when they combine a particular hook with the
use of pkgutil, to write the glue.

   pkgutil.register_hook(whatever)

Hook authors can, of course, implement a convenience method,
to be used as

   pkgutil.register_hook(ihook.pkgutil_hook)

but if they fail to, at least the user isn't left up the creek
without the proverbial paddle :-)

(Hope that made sense - I was hand-waving a bit too much...)

> > I don't think that either of these is even remotely a
> > showstopper, but they do need to be documented - specifically
> > because as a module, this code can be imported by anybody,
> > and so the constraints on the user should be made clear.
>
> Of course. Though with the current state of documentation,
> I give less priority to documenting every little detail of
> this code than to getting it in the code base (with *some*
> documentation, for sure).

At the moment, I'm playing with the hook mechanism, with the
intent of being able to provide some of the documentation. At
the very least, I'd like to be able to do a tutorial-style "how
to write a simple import hook" document.

But yes, a lot of this area is only minimally documented right
now, so we don't need to aim for complete perfection just yet :-)

> OTOH I expect that most things you'd want to place on sys.path
> *do* have a natural string representation. After all, if
> something doesn't have a name, it's hard to talk about. And if
> it has a name, it can be placed in PYTHONPATH (with the minor
> quibble that URLs use ':' which -- at least on Unix -- is also
> used as the separator in *PATH variables. :-( )

Hmm. Arguable. I take your point, but I'm looking at it from a
different point of view. A "place to store modules" could be
something as complex as a database (TCL Starkits use MetaKit).
I *could* define a string representation encoding the table name,
columns to use, etc etc, but why bother? Just use a database
importer object directly. You can't put it on PYTHONPATH, but
you can easily use something like this in a packaged application
context.

Let's wait and see what real use brings on this one.

> Maybe importer objects should have a standard API that lets you
> ask for an extended object.

This way lies a VFS implementation. I'd say we should write a
general import hook based on a VFS implementation, rather than the
other way around. And let's leave a full VFS alone for now :-)

Paul.


From barry@zope.com  Mon Dec 16 17:08:56 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 16 Dec 2002 12:08:56 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
References: <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com>
 <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com>
 <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
 <200212161654.gBGGsRe03537@odiug.zope.com>
Message-ID: <15870.2216.65988.322683@gargle.gargle.HOWL>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> BTW, here's a more recent draft, incorporating a search for
    GvR> .pth files and with a different place for the isdir():

Minor nit: .pth semantics should be identical to site.py .pth
semantics.  I.e. lines that start with "import" should be exec'd.

-Barry


From guido@python.org  Mon Dec 16 17:18:35 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 12:18:35 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 12:02:28 EST."
 <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com>
References: <Your message of "Mon, 16 Dec 2002 11:23:47 EST." <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
 <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com>
Message-ID: <200212161718.gBGHIZL03752@odiug.zope.com>

> Alternately, now that there's a "pkgutils" to put it in, why not add a 
> "fileNearModule()" function that returns a file handle opened for reading 
> on the basis of a module name and a file name?  I currently use something 
> roughly like:
> 
> def fileNearModule(moduleName,filename):
>      basedir = os.path.dirname(sys.modules[moduleName].__file__)
>      return open(os.path.join(basedir,filename), 'r')
> 
> Of course, a zipfile version would need some work, and clearly the zipfile 
> version wouldn't be a candidate for alpha 1.
> 
> Anyway, if we could establish an API for some of these sorts of things in 
> pkgutils, then they could be the official correct way to get at data files 
> and make zipfiles work for a lot more things long term.

I think Zope3 ought to defines its own API for this, then we can
decide later on how to do this.

I've seen several places where what you propose isn't enough: when I
tried putting everything in a zipfile, I ended up giving up when I
realized that there are places that use os.stat() on filenames thus
constructed.

Another issue is that, at least for zipfiles, and probably in most
realistic cases, the underlying implementation makes it easy to give
you the entire blob of data, and trying to pretend it's a file just
require wrapping it in a StringIO instance -- maybe it makes more
sense to change your fileNearModule() API to return the blob of data
too.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From aahz@pythoncraft.com  Mon Dec 16 17:36:42 2002
From: aahz@pythoncraft.com (Aahz)
Date: Mon, 16 Dec 2002 12:36:42 -0500
Subject: [Python-Dev] Deprecate doctest?
In-Reply-To: <200212161624.gBGGO2j01562@odiug.zope.com>
References: <BA1FF2F9.2D123%goodger@python.org> <3DFB101D.6010306@lemburg.com> <20021214194215.GA9536@panix.com> <200212161624.gBGGO2j01562@odiug.zope.com>
Message-ID: <20021216173642.GA10468@panix.com>

On Mon, Dec 16, 2002, Guido van Rossum wrote:
>Aahz:
>>
>> That's tricky, given that doctest is in the standard library, and that's
>> exactly what doctest does.  Are you advocating deprecating doctest?
> 
> You gotta be kidding.

Of *course* I was, but I couldn't figure out what else MAL meant.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From guido@python.org  Mon Dec 16 17:47:34 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 12:47:34 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 12:08:56 EST."
 <15870.2216.65988.322683@gargle.gargle.HOWL>
References: <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <200212161654.gBGGsRe03537@odiug.zope.com>
 <15870.2216.65988.322683@gargle.gargle.HOWL>
Message-ID: <200212161747.gBGHlYK03859@odiug.zope.com>

> Minor nit: .pth semantics should be identical to site.py .pth
> semantics.  I.e. lines that start with "import" should be exec'd.

Barry & I had an offline discussion about this.

Some conclusions:

- pth files for sys.path and pth-ish files for packages should
  probably have different extensions; otherwise e.g. paths named in
  zope.pth would be added both to sys.path and to zope.__path__, which
  is a bad idea.  I propose using pkg as the extension for package.s

- Nobody remembers why there is a special case in site.py for lines in
  the .pth file that start with 'import', except that Martin von
  Loewis once wanted this.  I propose not to duplicate this
  functionality in .pkg files.

- We need to look both for subdirectories and for .pkg files --
  different styles of administering a system call for both approaches.

- We should only accept subdirectories that have an __init__.py.

- I'm punting for now on an extension to cope with finding things in
  other importers.

Current draft, incorporating all these changes:

######################################################################
"""Utilities to support packages."""

import os
import sys

def extend_path(path, name):
    """Extend a package's path.

    Intended use is to place the following code in a package's __init__.py:

        from pkgutil import extend_path
        __path__ = extend_path(__path__, __name__)

    This will add to the package's __path__ all subdirectories of
    directories on sys.path named after the package.  This is useful
    if one wants to distribute different parts of a single logical
    package as multiple directories.

    It also looks for *.pkg files beginning with __name__.  This
    feature is similar to *.pth files, except it doesn't special-case
    lines starting with 'import'.
    """
    pname = os.path.join(*name.split('.')) # Reconstitute as relative path
    sname = os.extsep.join(name.split('.')) # Just in case os.extsep != '.'
    sname_pkg = sname + os.extsep + "pkg"
    init_py = "__init__" + os.extsep + "py"
    path = path[:]
    for dir in sys.path:
        if isinstance(dir, (str, unicode)) and os.path.isdir(dir):
            subdir = os.path.join(dir, pname)
            # XXX This may still add duplicate entries to path on
            # case-insensitive filesystems
            initfile = os.path.join(subdir, init_py)
            if subdir not in path and os.path.isfile(initfile):
                path.append(subdir)
            # XXX Is this the right thing for subpackages like
            # zope.app?  It looks for a file named "zope.app.pkg"
            pkgfile = os.path.join(dir, sname_pkg)
            if os.path.isfile(pkgfile):
                try:
                    f = open(pkgfile)
                except IOError, msg:
                    sys.stderr.write("Can't open %s: %s\n" %
                                     (pkgfile, msg))
                else:
                    for line in f:
                        line = line.rstrip('\n')
                        if not line or line.startswith('#'):
                            continue
                        path.append(line) # Don't check for existence!
                    f.close()
    return path
######################################################################

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@zope.com  Mon Dec 16 17:59:31 2002
From: jim@zope.com (Jim Fulton)
Date: Mon, 16 Dec 2002 12:59:31 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re:  directory hierarchy proposal)
References: <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
Message-ID: <3DFE1483.6010703@zope.com>

Phillip J. Eby wrote:
> (Summary: good plan for name and place, issues exist w/sketched 
> implementation)
> 
> At 10:47 AM 12/16/02 -0500, Guido van Rossum wrote:
> 
>> Time's running short (we're close to releasing Python 2.3 alpha 1);
>> we should decide on a name and a specific policy ASAP.  I propose the
>> following:
>>
>>   from pkgutil import extended_path
>>   __path__ = extended_path(__path__, __name__)
> 
> 
> How would that work for say, 'zope.app'?

Since zope.app isn't a namespace package, it doesn't need to.

But I suppose that the question stands for zodb.storages. :)

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (888) 344-4332            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org



From jim@zope.com  Mon Dec 16 18:04:00 2002
From: jim@zope.com (Jim Fulton)
Date: Mon, 16 Dec 2002 13:04:00 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
References: <Your message of "Mon, 16 Dec 2002 11:23:47 EST." <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>              <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com>
Message-ID: <3DFE1590.50908@zope.com>

Guido van Rossum wrote:
>>Alternately, now that there's a "pkgutils" to put it in, why not add a 
>>"fileNearModule()" function that returns a file handle opened for reading 
>>on the basis of a module name and a file name?  I currently use something 
>>roughly like:
>>
>>def fileNearModule(moduleName,filename):
>>     basedir = os.path.dirname(sys.modules[moduleName].__file__)
>>     return open(os.path.join(basedir,filename), 'r')
>>
>>Of course, a zipfile version would need some work, and clearly the zipfile 
>>version wouldn't be a candidate for alpha 1.
>>
>>Anyway, if we could establish an API for some of these sorts of things in 
>>pkgutils, then they could be the official correct way to get at data files 
>>and make zipfiles work for a lot more things long term.
>>
> 
> I think Zope3 ought to defines its own API for this, then we can
> decide later on how to do this.
> 
> I've seen several places where what you propose isn't enough: when I
> tried putting everything in a zipfile, I ended up giving up when I
> realized that there are places that use os.stat() on filenames thus
> constructed.

This is to ease development. In development mode, we check if the
data on disk has changed and reread it if it has. This wouldn't be
applicable to zip files.

Jim


-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (888) 344-4332            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org



From fredrik@pythonware.com  Mon Dec 16 18:06:32 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 16 Dec 2002 19:06:32 +0100
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
References: <16E1010E4581B049ABC51D4975CEDB88619987@UKDCX001.uk.int.atosorigin.com> <200212161642.gBGGgYJ03474@odiug.zope.com>              <15870.939.16954.763349@gargle.gargle.HOWL>  <200212161700.gBGH02S03607@odiug.zope.com>
Message-ID: <00d601c2a52d$e2c9c2e0$ced241d5@hagrid>

Guido van Rossum wrote:


> > Or even extendpath() <wink>>
> 
> Jim doesn't like sequences of lowercase words strung together without
> using either camelcase or underscores.

isn't Jim using Python?

>>> dir("")
['capitalize', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find',
'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper',
'join', 'ljust', 'lower', 'lstrip', 'replace', 'rfind', 'rindex', 'rjust', 'rstrip',
'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate',
'upper']
>>> dir(os.path)
[... 'abspath', 'basename', 'commonprefix', 'dirname', 'exists', 'expanduser',
'expandvars', 'getatime ', 'getmtime', 'getsize', 'isabs', 'isdir', 'isfile', 'islink',
'ismount', 'join', 'normcase', 'normpath', 'os', 'split', 'splitdrive', 'splitext',
'splitunc', 'stat', 'walk']
>>> dir({})
[..., 'clear', 'copy', 'get', 'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues',
'keys', 'pop', 'popitem', 'setdefault', 'update', 'values']
>>> dir(sys)
[..., '_getframe', 'api_version', 'argv', 'builtin_module_names', 'byteorder',
'copyright', 'displayhook', 'exc_info', 'exc_type', 'excepthook', 'exec_prefix',
'executable', 'exit', 'getdefaultencoding', 'getdlopenflags', 'getrecursionlimit',
'getrefcount', 'hexversion', 'last_traceback', 'last_type', 'last_value', 'maxint',
'maxunicode', 'modules', 'path', 'platform', 'prefix', 'ps1', 'ps2',
'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit',
'settrace', 'stderr', 'stdin', 'stdout', 'version', 'version_info',
'warnoptions']

if I didn't know better, I'd surely think that sequences of lowercase words
strung together without underscores is the standard way to name functions
and methods in Python...

</F>



From guido@python.org  Mon Dec 16 18:08:38 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 13:08:38 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 13:04:00 EST."
 <3DFE1590.50908@zope.com>
References: <Your message of "Mon, 16 Dec 2002 11:23:47 EST." <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com>
 <3DFE1590.50908@zope.com>
Message-ID: <200212161808.gBGI8cP04069@odiug.zope.com>

> > I've seen several places where what you propose isn't enough: when I
> > tried putting everything in a zipfile, I ended up giving up when I
> > realized that there are places that use os.stat() on filenames thus
> > constructed.
> 
> This is to ease development. In development mode, we check if the
> data on disk has changed and reread it if it has. This wouldn't be
> applicable to zip files.

How do we turn this off?  I think it would be a good exercise to see
if we could load Zope3 from a zip file.  (Though I've got to say that
the ~3 seconds startup time on my 2 year old Linux box is pretty
speedy, so I'm not sure we could or need to make it much faster than
that.)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@zope.com  Mon Dec 16 18:24:56 2002
From: jim@zope.com (Jim Fulton)
Date: Mon, 16 Dec 2002 13:24:56 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
References: <Your message of "Mon, 16 Dec 2002 11:23:47 EST." <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com>              <3DFE1590.50908@zope.com> <200212161808.gBGI8cP04069@odiug.zope.com>
Message-ID: <3DFE1A78.3080004@zope.com>

Guido van Rossum wrote:
>>>I've seen several places where what you propose isn't enough: when I
>>>tried putting everything in a zipfile, I ended up giving up when I
>>>realized that there are places that use os.stat() on filenames thus
>>>constructed.
>>>
>>This is to ease development. In development mode, we check if the
>>data on disk has changed and reread it if it has. This wouldn't be
>>applicable to zip files.
>>
> 
> How do we turn this off? 

In the zconfig file. ;)

There isn't a way to turn it off yet.

 > I think it would be a good exercise to see
> if we could load Zope3 from a zip file.  (Though I've got to say that
> the ~3 seconds startup time on my 2 year old Linux box is pretty
> speedy, so I'm not sure we could or need to make it much faster than
> that.)

I presume zip files would make mac starup a lot faster, but that
wouldn't work for development.  It would also help startup
when reading from CD, or so the theory goes. Having said all of that,
I have a hard time getting excited about zip file import myself.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (888) 344-4332            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org



From guido@python.org  Mon Dec 16 18:31:01 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 13:31:01 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 13:24:56 EST."
 <3DFE1A78.3080004@zope.com>
References: <Your message of "Mon, 16 Dec 2002 11:23:47 EST." <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <Your message of "Mon, 16 Dec 2002 10:15:01 EST." <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <20021210103937.GA21447@vet.uu.nl> <15862.10111.488361.591077@gargle.gargle.HOWL> <20021210191501.GC23933@vet.uu.nl> <15862.16384.979878.277669@gargle.gargle.HOWL> <20021211083643.GB25606@vet.uu.nl> <15863.10433.412101.910639@gargle.gargle.HOWL> <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl> <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de> <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com> <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com> <5.1.1.6.0.20021216115554.02c75ec0@telecommunity.com> <200212161718.gBGHIZL03752@odiug.zope.com> <3DFE1590.50908@zope.com> <200212161808.gBGI8cP04069@odiug.zope.com>
 <3DFE1A78.3080004@zope.com>
Message-ID: <200212161831.gBGIV1i04265@odiug.zope.com>

> I presume zip files would make mac starup a lot faster, but that
> wouldn't work for development.

Except if you can update the zipfile is under 25 seconds. ;-)

> It would also help startup when reading from CD, or so the theory
> goes. Having said all of that, I have a hard time getting excited
> about zip file import myself.

Agreed.


From =?iso-8859-1?Q?Martin_v._L=F6wis?=  Mon Dec 16 18:33:32 2002
From: =?iso-8859-1?Q?Martin_v._L=F6wis?= (=?iso-8859-1?Q?Martin_v._L=F6wis?=)
Date: Mon, 16 Dec 2002 19:33:32 +0100
Subject: [Python-Dev] PEP-298:  buffer interface widening too much?
References: <3DFCCAFD.6020303@stsci.edu> <017301c2a46d$7ade6160$d435e8d9@mira>	<3DFCFD21.2090707@stsci.edu> <m3fzsylttd.fsf@mira.informatik.hu-berlin.de> <3DFDE30C.7080009@stsci.edu>
Message-ID: <017d01c2a531$a34083d0$9319e8d9@mira>

> >We could even provide a Generic function which
> >delegates the new get functions to the existing ones, accessing
> >segment zero.
> >  
> >
> Wouldn't this require more information about the lock implementation? 

The generic function would just invoke the unlocked routine, and thus
it would only be applicable for types that don't require locking (as
the underlying memory can't move, anyway). If there must be a true
lock count in the implementation, you couldn't use the generic wrapper.

I don't know how useful that would be, as I don't know whether the typical
buffer-supporting object has movable memory or fixed memory; I'd expect that
immutable object typically have fixed memory, and mutable objects typically
have relocatable memory. So only immutable types could use the wrapper.

>  As PEP-298 stands,  I thought that was left up to the buffered type. 
>  It might make sense to change the locking getpointer fields into lock, 
> and release into unlock.  Then compose the primitive operations in the 
> wrapper as you suggested.   That wouldn't fix the size_t problem though.

Ah, right - this would be another restriction for using the generic wrapper.
Perhaps then there is really no point in having it.

> >I now recall what the issue is: you can't assume that modifying the
> >argument tuple object to record resources is feasible, as this breaks
> >apply (where in every case the user allocates a tuple, and where this
> >is a plain old tuple).
> >  
> >
> Is the efficiency of apply() so critical that a conversion is ruled out?

I would think so, yes: allocating objects is one of the most costly operation
in Python. It is more efficient now that we have pymalloc, but still.

There is also the in-C calls to functions that use Py_BuildValue, then
PyObject_Call; returning an enhanced tuple from BuildValue might break
existing code, yet copying the tuples inside PyObject_Call (if needed)
would add a price there, too.

So I'm now leaning towards not requiring a new type.

> Way over my head here...  I guess the point is that arg tuples aren't 
> the only point of attack.

No, they really are the issue. However, I cannot guarantee that ParseTuple
is always called with enhanced tuples, so the correctness this is meant to
provide isn't really provided.

Regards,
Martin



From just@letterror.com  Mon Dec 16 19:14:36 2002
From: just@letterror.com (Just van Rossum)
Date: Mon, 16 Dec 2002 20:14:36 +0100
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: <200212161718.gBGHIZL03752@odiug.zope.com>
Message-ID: <r01050400-1022-BD2FA884112A11D7AD06003065D5E7E4@[10.0.0.23]>

FWIW, I think it's generally a bad idea to depend on module.__file__ and
pkg.__path__ to find data files. Playing tricks with the contents of
pkg.__path__ apparently has its uses, but I think that (in general) it's a bad
idea as well. module.__file__ is mostly an introspective aid, and pkg.__path__
should IMO be seen as merely an implementation detail.

module.__file__: in a frozen module it will be set to "<frozen>". Expect
__file__ to be a path to a file and you're screwed.

To the lower levels of the import mechanism, the *existence* of pkg.__path__ is
all that's looked at: "hey, it's a package". Then there's freeze again: a frozen
package has a __path__ variable, but it's not a list, it's a *string*. Only when
an import goes through a sys.path item, __path__ is (more or less) guaranteed to
be a list.

The sys.meta_path import hook mechanism in my patch (the idea is stolen from
Gordon McMillan) acts on the same level as builtin module imports and frozen
module imports: it doesn't need sys.path. So it doesn't need any meaningful
object as pkg.__path__ either. I just uploaded a new version of the patch; it
now contains a test_importhooks.py script, which has a sys.meta_path test case
which actually sets pkg.__path__ to None. Works like a charm. Here's a (slightly
modified) comment from the test script:

    Depending on the kind of importer, there are different
    levels of freedom of what you can use as pkg.__path__.
    
    Importer object on sys.meta_path:
        it can use anything it pleases (even None), as long
        as a __path__ variable is set.
    Importer object on sys.path:
        pkg.__path__ must be a list; it's most logical to use
        an importer object as the only item. Could be the same
        importer instance that imported the package itself.
    A hook on sys.path_hooks:
        pkg.__path__ must be a list and its only item should
        be a string that the hook can handle itself.
    
    These are just guidelines: a set of hooks could in theory
    deliberately set pgk.__path__ up so submodule imports be
    handled by an entirely different importer. Not sure how
    useful that would be...

Just-my-2-eurocents.


From niemeyer@conectiva.com  Mon Dec 16 19:19:36 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Mon, 16 Dec 2002 17:19:36 -0200
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <200212161631.gBGGVFB01638@odiug.zope.com>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com>
Message-ID: <20021216171936.A10545@ibook.distro.conectiva>

> Hm...  Do you see any way to break out of restricted execution mode
> using this?  I suppose a fix would be simple enough, but I'm more and
> more inclined to simply rip out rexec from the distribution -- it's
> never going to be safe, and I doubt it's very useful as long as it's
> not safe.

Do you mean the rexec.py module, or all the restricted features?

I would like to see the whole scheme working someday. I'm not sure how
safe it will ever be, but the problems I've seen so far are due to some
language change introduced. I belive that once the language features are
mature, these problems will be reduced.

As a side note, I'll have a look at that bug, if nobody has done so yet.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From theller@python.net  Mon Dec 16 19:22:01 2002
From: theller@python.net (Thomas Heller)
Date: 16 Dec 2002 20:22:01 +0100
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: <r01050400-1022-BD2FA884112A11D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-BD2FA884112A11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <pts17orq.fsf@python.net>

Just van Rossum <just@letterror.com> writes:

> FWIW, I think it's generally a bad idea to depend on module.__file__ and
> pkg.__path__ to find data files.
What would you suggest to do instead? There's now even such a case
in the standard library: distutils.command.bdist_wininst has
to find it's data-file 'wininst.exe'.

Thomas



From guido@python.org  Mon Dec 16 19:22:27 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 14:22:27 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 17:05:28 GMT."
 <16E1010E4581B049ABC51D4975CEDB88619988@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619988@UKDCX001.uk.int.atosorigin.com>
Message-ID: <200212161922.gBGJMRe05832@odiug.zope.com>

> > > (and of course, objects other than string subclasses don't
> > > get "extended").
> >
> > We can use some kind of extension hook to deal with those.
> 
> It depends where you envisage that hook. I'd be unhappy with
> requiring something in the import hook implementation - two
> reasons, first because the current hook mechanism is nice and
> simple, and hence easy to document and use, and second because
> people simply won't implement the path-extension hook, and nearly
> everything will work fine - the breakage only occurs when the
> hook is used in conjunction with pkgutil. At that point, saying
> "I told you so" isn't much help...
> 
> I've seen this sort of thing happen too often before to be
> comfortable with it. A simple protocol, with an obscure little
> requirement which is to support a relatively rare usage. People
> *never* implement that correctly, and the result is annoying
> breakages.

Only annoying if you need the rare usage.  It turns out that running
from a zipfile isn't a priority for Zope3 anyway, so I'll call this a
YAGNI.

> > The important point is that the idiom placed in __init__.py
> > can remain the same (the two lines quoted above) while the
> > implementation of pkgutil.extend_path() can evolve or be
> > modified (even monkey-patched if you're desperate) as the
> > import hooks evolve.
> 
> I'd say the extension mechanism should be external to both
> the import hook and pkgutil. It then becomes the user's
> responsibility, when they combine a particular hook with the
> use of pkgutil, to write the glue.
> 
>    pkgutil.register_hook(whatever)
> 
> Hook authors can, of course, implement a convenience method,
> to be used as
> 
>    pkgutil.register_hook(ihook.pkgutil_hook)
> 
> but if they fail to, at least the user isn't left up the creek
> without the proverbial paddle :-)
> 
> (Hope that made sense - I was hand-waving a bit too much...)

Yes, except you failed to specify what pkgutil.register_hook(whatever)
is supposed to do, and you failed to specify the signature for
'whatever'...  IOW, please think more about the implementation and
provide a patch to the latest pkgutil.py (I posted it here earlier).

> > > I don't think that either of these is even remotely a
> > > showstopper, but they do need to be documented - specifically
> > > because as a module, this code can be imported by anybody,
> > > and so the constraints on the user should be made clear.
> >
> > Of course. Though with the current state of documentation,
> > I give less priority to documenting every little detail of
> > this code than to getting it in the code base (with *some*
> > documentation, for sure).
> 
> At the moment, I'm playing with the hook mechanism, with the
> intent of being able to provide some of the documentation. At
> the very least, I'd like to be able to do a tutorial-style "how
> to write a simple import hook" document.
> 
> But yes, a lot of this area is only minimally documented right
> now, so we don't need to aim for complete perfection just yet :-)

You're talking about Just's import hooks, right?

> > OTOH I expect that most things you'd want to place on sys.path
> > *do* have a natural string representation. After all, if
> > something doesn't have a name, it's hard to talk about. And if
> > it has a name, it can be placed in PYTHONPATH (with the minor
> > quibble that URLs use ':' which -- at least on Unix -- is also
> > used as the separator in *PATH variables. :-( )
> 
> Hmm. Arguable. I take your point, but I'm looking at it from a
> different point of view. A "place to store modules" could be
> something as complex as a database (TCL Starkits use MetaKit).
> I *could* define a string representation encoding the table name,
> columns to use, etc etc, but why bother? Just use a database
> importer object directly. You can't put it on PYTHONPATH, but
> you can easily use something like this in a packaged application
> context.

A packaged application knows whether it needs to support pkgutil or
not.  So it can take whatever measures necessary (including overriding
pkgutil.py) to make it work.

> Let's wait and see what real use brings on this one.

Yes please.

> > Maybe importer objects should have a standard API that lets you
> > ask for an extended object.
> 
> This way lies a VFS implementation.

I'm not sure what you mean by that.

> I'd say we should write a general import hook based on a VFS
> implementation, rather than the other way around. And let's leave a
> full VFS alone for now :-)

The old ihooks was trying to do something like a vfs (it has a bunch
of APIs that provide hooks into all filesystem operations).  But it
wasn't a big success...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From shane@zope.com  Mon Dec 16 19:37:42 2002
From: shane@zope.com (Shane Hathaway)
Date: Mon, 16 Dec 2002 14:37:42 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation
 (was Re: directory hierarchy proposal)
References: <r01050400-1022-BD2FA884112A11D7AD06003065D5E7E4@[10.0.0.23]> <pts17orq.fsf@python.net>
Message-ID: <3DFE2B86.3060508@zope.com>

Thomas Heller wrote:
> Just van Rossum <just@letterror.com> writes:
> 
> 
>>FWIW, I think it's generally a bad idea to depend on module.__file__ and
>>pkg.__path__ to find data files.
> 
> What would you suggest to do instead? There's now even such a case
> in the standard library: distutils.command.bdist_wininst has
> to find it's data-file 'wininst.exe'.

I'd like to point out Java's solution to this.

http://java.sun.com/j2se/1.4.1/docs/api/java/lang/ClassLoader.html#getResource(java.lang.String)

Since Python doesn't have the complication of multiple class loaders, 
getResource() (or whatever name it gets in Python) and its peers could 
be module-level functions somewhere (such as pkgutils).

Shane



From guido@python.org  Mon Dec 16 19:41:55 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 14:41:55 -0500
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: Your message of "Mon, 16 Dec 2002 20:14:36 +0100."
 <r01050400-1022-BD2FA884112A11D7AD06003065D5E7E4@[10.0.0.23]>
References: <r01050400-1022-BD2FA884112A11D7AD06003065D5E7E4@[10.0.0.23]>
Message-ID: <200212161941.gBGJftc05924@odiug.zope.com>

[Please try to limit the width of your lines to 72 characters or so.]

> FWIW, I think it's generally a bad idea to depend on module.__file__
> and pkg.__path__ to find data files.

Of course you think this, because you just recently invented something
that would break this assumption. :-)

The problem is that Zope3 needs *some* way of storing data and config
files related to specific packages, and there isn't really any other
convenient and conventional way except storing it in a package
directory and using the package's __file__ to find it.

Python's own test package uses this, and so does the pdb module for
its help file (the latter actually scans sys.path for a file pdb.doc).

> Playing tricks with the contents of pkg.__path__ apparently has its
> uses,

It's the *defined* API for adding to the set of locations from which
submodules and subpackages of a package are to be loaded.  You can't
just call that "playing tricks."

> but I think that (in general) it's a bad idea as
> well. module.__file__ is mostly an introspective aid, and
> pkg.__path__ should IMO be seen as merely an implementation detail.

Not true, see above -- it's a defined API for a specific purpose.

That's less so for __file__, but absent a well-defined alternative
API, __file__ should be supported.

I'm not saying that __file__ should always point to a file --
obviously it can't, if we want import from zip files.  But there's
considerable existing code that uses it in certain ways and we
shouldn't break that unless necessary.

> module.__file__: in a frozen module it will be set to
> "<frozen>". Expect __file__ to be a path to a file and you're
> screwed.

Of course.  If you freeze code, you have to make sure that it will be
able to find its data files in some other way.  But not all code has
to be freezable.

> To the lower levels of the import mechanism, the *existence* of
> pkg.__path__ is all that's looked at: "hey, it's a package". Then
> there's freeze again: a frozen package has a __path__ variable, but
> it's not a list, it's a *string*. Only when an import goes through a
> sys.path item, __path__ is (more or less) guaranteed to be a list.

Yeah, frozen code can't work with pkgutil as it currently stands.  But
I think pkgutil is pretty meaningless for frozen code: pkgutil is a
mechanism to allow extensions of a package to be installed in
different directories.  A frozen application should gather all
code belonging to a package in a single directory, and freeze that;
then all we need to ensure is that calling pkgutil.extend_path()
doesn't bomb out, and that's easily done by adding

  if not isinstance(path, list):
      return path

to the start of extend_path().

> The sys.meta_path import hook mechanism in my patch (the idea is
> stolen from Gordon McMillan) acts on the same level as builtin
> module imports and frozen module imports: it doesn't need
> sys.path. So it doesn't need any meaningful object as pkg.__path__
> either. I just uploaded a new version of the patch; it now contains
> a test_importhooks.py script, which has a sys.meta_path test case
> which actually sets pkg.__path__ to None. Works like a charm. Here's
> a (slightly modified) comment from the test script:

Does the metahook also apply to submodules (or subpackage) of
packages?  I'd expect not.  (I haven't had the time to review your
latest patches yet, in part due to this thread. :-)  Surely you
shouldn't be looking for builtin submodules of a package.

>     Depending on the kind of importer, there are different
>     levels of freedom of what you can use as pkg.__path__.
>     
>     Importer object on sys.meta_path:
>         it can use anything it pleases (even None), as long
>         as a __path__ variable is set.

But I can imagine that some metahooks would like to look inside the
__path__ list for more hints on where to find the submodules of the
package.  Of course that's up to the metahook.

>     Importer object on sys.path:
>         pkg.__path__ must be a list; it's most logical to use
>         an importer object as the only item. Could be the same
>         importer instance that imported the package itself.

Multiple items would make sense too.

>     A hook on sys.path_hooks:
>         pkg.__path__ must be a list and its only item should
>         be a string that the hook can handle itself.

IMO it should allow multiple strings too!

>     These are just guidelines: a set of hooks could in theory
>     deliberately set pgk.__path__ up so submodule imports be
>     handled by an entirely different importer. Not sure how
>     useful that would be...

Very useful, for not-yet-imagined cases.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Mon Dec 16 19:47:34 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 14:47:34 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: Your message of "Mon, 16 Dec 2002 17:19:36 -0200."
 <20021216171936.A10545@ibook.distro.conectiva>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com>
 <20021216171936.A10545@ibook.distro.conectiva>
Message-ID: <200212161947.gBGJlYA05956@odiug.zope.com>

> > Hm...  Do you see any way to break out of restricted execution mode
> > using this?  I suppose a fix would be simple enough, but I'm more and
> > more inclined to simply rip out rexec from the distribution -- it's
> > never going to be safe, and I doubt it's very useful as long as it's
> > not safe.
> 
> Do you mean the rexec.py module, or all the restricted features?

Both.  Unless we spend several orders of more effort on reviewing and
testing these "security" features, we're running the serious risk that
someone naively believes that they are secure, uses them to protect
real data, and their site gets broken into.  Then who is responsible?
Even if no real data is lost, the more we advertise this as secure,
the more egg we have on our face when someone finds a hole.  The
history of Java's security features shows that even in systems that
have had infinitely more scrutiny, security holes still show up --
a language implementation is simply too complex to be bug-free.

> I would like to see the whole scheme working someday. I'm not sure
> how safe it will ever be, but the problems I've seen so far are due
> to some language change introduced. I belive that once the language
> features are mature, these problems will be reduced.

I wish you well, but I recommend that you start a separate project
"secure Python".  I don't think that the core will ever slow down its
evolution to a pace where security issues can be fixed faster than
they are generated by new code.

> As a side note, I'll have a look at that bug, if nobody has done so yet.

Please do.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pedronis@bluewin.ch  Mon Dec 16 19:59:41 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Mon, 16 Dec 2002 20:59:41 +0100
Subject: [Python-Dev] known obvious thing or bug (rexec)?
References: <015b01c2a385$97010700$6d94fea9@newmexico>  <200212161631.gBGGVFB01638@odiug.zope.com>
Message-ID: <002801c2a53d$ac854b40$6d94fea9@newmexico>

From: "Guido van Rossum" <guido@python.org>
> > latest CVS
> >
> > Python 2.3a0 (#1, Dec 14 2002, 07:19:32)
> > [GCC 2.95.2 20000220 (Debian GNU/Linux)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >
> > consider
> >
> > >>> import rexec
> > >>> r=rexec.RExec()
> > >>> r.r_exec("class C: pass\nC.a=0")
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> >   File "/home/users/p/pe/pedronis/python/dist/src/Lib/rexec.py", line 309,
in
> > r_
> > exec
> >     exec code in m.__dict__
> >   File "<string>", line 2, in ?
> > RuntimeError: classes are read-only in restricted mode
> >
> > vs.
> >
> > >>> r.r_exec("class C(object): pass\nC.a=0")
> > >>>
>
> Hm...  Do you see any way to break out of restricted execution mode
> using this?  I suppose a fix would be simple enough, but I'm more and
> more inclined to simply rip out rexec from the distribution -- it's
> never going to be safe, and I doubt it's very useful as long as it's
> not safe.

basically all the checks (inaccessible __dict__, readonly __class__ etc)  done
for old-style classes are not done for new-style classes,
things get also complicated because of metaclasses (one could enforce things in
type but ... )

I agree (but I have said that in past) the best thing is to deprecate/rip out
rexec.
I have looked a bit at zope.security, it seems bases on a better model, maybe
if it's successful it could be incorparated in the std lib in the future ...

regards.





From guido@python.org  Mon Dec 16 20:09:39 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 15:09:39 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: Your message of "Mon, 16 Dec 2002 20:59:41 +0100."
 <002801c2a53d$ac854b40$6d94fea9@newmexico>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com>
 <002801c2a53d$ac854b40$6d94fea9@newmexico>
Message-ID: <200212162009.gBGK9dI06095@odiug.zope.com>

> I agree (but I have said that in past) the best thing is to
> deprecate/rip out rexec.  I have looked a bit at zope.security, it
> seems bases on a better model, maybe if it's successful it could be
> incorparated in the std lib in the future ...

But of course to some extent Zope's security depends on some of the
features of Python's security (like being able to override
__builtins__).

Though especially the Zope3 approach, using security proxies
implemented in C, seems quite safe: *no* attribute can be accessed
without confirmation from the security checker.  In this situation,
type(x) will returned a security proxy for the type of x; type is
replaced with a security-aware version when executing untrusted code.

I note that we're finding it convenient to let __str__ and __repr__
bypass the security proxy though.  This means that str() of a
security-proxy-wrapped container will call str() on the unwrapped
items inside the container.  Of course, this simply means that __str__
and __repr__ should not reveal sensitive information; not a big burden
IMO.

Disclaimer: I know shit about security. :-(

--Guido van Rossum (home page: http://www.python.org/~guido/)


From sholden@holdenweb.com  Mon Dec 16 20:12:24 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Mon, 16 Dec 2002 15:12:24 -0500
Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python
References: <200212142106.01812.mclay@nist.gov>  <200212161627.gBGGRq801621@odiug.zope.com>
Message-ID: <010a01c2a53f$75281900$6b01a8c0@holdenweb.com>

[Micheal McLay]
> > sf/653938I is a patch to integrate the fixedpoint module, that was
> > created by Tim Peters, into Python as a first class object. That is,
> > the patch adds this new number type so that it has direct syntax
> > support like float, int, long, str., etc. within the interpreter.

[Guido]
>
> (a) I don't want any (more) change to the syntax for Python 2.3.
>
> (b) Like the sets module, proposals for new datatypes should first be
>     prototyped as standard library modules before being promoted to
>     built-in status (with their own literals).
>
Speaking of which, if sets *were* to become built-ins, would the notation

    { member1, member2, member3, ... }

fly as a constructor literal? I don't know enough ab out the parser's
operations to determine how easy disambiguation from a dictionary
constructor might be, but it's certainly a desirably parallel notation with
mathematical sets.

all-a-long-way-from-2.3-ly y'rs  - steve
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Bring your musical instrument to PyCon!    http://www.python.org/pycon/
-----------------------------------------------------------------------





From guido@python.org  Mon Dec 16 20:18:49 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 15:18:49 -0500
Subject: [Python-Dev] Adding decimal (aka FixedPoint) numbers to Python
In-Reply-To: Your message of "Mon, 16 Dec 2002 15:12:24 EST."
 <010a01c2a53f$75281900$6b01a8c0@holdenweb.com>
References: <200212142106.01812.mclay@nist.gov> <200212161627.gBGGRq801621@odiug.zope.com>
 <010a01c2a53f$75281900$6b01a8c0@holdenweb.com>
Message-ID: <200212162018.gBGKIos06623@odiug.zope.com>

> Speaking of which, if sets *were* to become built-ins, would the
> notation
> 
>     { member1, member2, member3, ... }
> 
> fly as a constructor literal? I don't know enough ab out the
> parser's operations to determine how easy disambiguation from a
> dictionary constructor might be, but it's certainly a desirably
> parallel notation with mathematical sets.

That's what PEP 218 proposes, yes (it even proposes "set
comprehensions").  The parser has no problem with this, except that {}
would be ambiguous.  The PEP proposes to solve that by using {-} to
write an empty set, but not everyone likes that.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From dave@boost-consulting.com  Mon Dec 16 20:57:54 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 16 Dec 2002 15:57:54 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <200212162009.gBGK9dI06095@odiug.zope.com> (Guido van Rossum's
 message of "Mon, 16 Dec 2002 15:09:39 -0500")
References: <015b01c2a385$97010700$6d94fea9@newmexico>
 <200212161631.gBGGVFB01638@odiug.zope.com>
 <002801c2a53d$ac854b40$6d94fea9@newmexico>
 <200212162009.gBGK9dI06095@odiug.zope.com>
Message-ID: <uu1hdel65.fsf@boost-consulting.com>

Guido van Rossum <guido@python.org> writes:

> Disclaimer: I know shit about security. :-(

At least you're ahead of some of us.  I don't know shit about
security.

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From niemeyer@conectiva.com  Mon Dec 16 21:02:02 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Mon, 16 Dec 2002 19:02:02 -0200
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <200212161947.gBGJlYA05956@odiug.zope.com>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com>
Message-ID: <20021216190202.A11682@ibook.distro.conectiva>

> > Do you mean the rexec.py module, or all the restricted features?
> 
> Both.  Unless we spend several orders of more effort on reviewing and
> testing these "security" features, we're running the serious risk that
> someone naively believes that they are secure, uses them to protect
> real data, and their site gets broken into.  Then who is responsible?

Perhaps the main features could be left in the core, without any real
application being offered as part of the "batteries". This way, a secure
environment could evolve in parallel, without forking the Python
development itself.

> Even if no real data is lost, the more we advertise this as secure,
> the more egg we have on our face when someone finds a hole.  The
> history of Java's security features shows that even in systems that
> have had infinitely more scrutiny, security holes still show up --
> a language implementation is simply too complex to be bug-free.

Agreed.

> > I would like to see the whole scheme working someday. I'm not sure
> > how safe it will ever be, but the problems I've seen so far are due
> > to some language change introduced. I belive that once the language
> > features are mature, these problems will be reduced.
> 
> I wish you well, but I recommend that you start a separate project
> "secure Python".  I don't think that the core will ever slow down its
> evolution to a pace where security issues can be fixed faster than
> they are generated by new code.

I was talking about major changes like type/class unification. I belive
that major changes like this won't happen often, and that's the kind of
change that affected the restricted execution so far.

One idea I was wondering is to provide a "hack me" machine somewhere on
the web, including a hall-of-fame for those who break the system
somehow. Hummm.. I must find some infrastructure to do that. Anyway,
that could help us find some obvious problems quickly, but I understand
it shouldn't affect the general decision of how to support restricted
execution in Python.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From bac@OCF.Berkeley.EDU  Mon Dec 16 21:05:38 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Mon, 16 Dec 2002 13:05:38 -0800 (PST)
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <uu1hdel65.fsf@boost-consulting.com>
References: <015b01c2a385$97010700$6d94fea9@newmexico>
 <200212161631.gBGGVFB01638@odiug.zope.com> <002801c2a53d$ac854b40$6d94fea9@newmexico>
 <200212162009.gBGK9dI06095@odiug.zope.com> <uu1hdel65.fsf@boost-consulting.com>
Message-ID: <Pine.SOL.4.51.0212161302530.2813@death.OCF.Berkeley.EDU>

[David Abrahams]

> Guido van Rossum <guido@python.org> writes:
>
> > Disclaimer: I know shit about security. :-(
>
> At least you're ahead of some of us.  I don't know shit about
> security.
>

Now if I quote  you two, should I censor it  for the possible little
children who will read this naughty language?  =)

trying-to-not-swear-like-on-the-soparnos-ly y'rs, Brett


From guido@python.org  Mon Dec 16 21:37:46 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 16:37:46 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: Your message of "Mon, 16 Dec 2002 19:02:02 -0200."
 <20021216190202.A11682@ibook.distro.conectiva>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com>
 <20021216190202.A11682@ibook.distro.conectiva>
Message-ID: <200212162137.gBGLbkl07474@odiug.zope.com>

> Perhaps the main features could be left in the core, without any real
> application being offered as part of the "batteries". This way, a secure
> environment could evolve in parallel, without forking the Python
> development itself.

Of course that assumes that the main features in the core are secure.
Samuele's observation that restricted code can modify a new-style
class passed in belies that.

> I was talking about major changes like type/class unification. I belive
> that major changes like this won't happen often, and that's the kind of
> change that affected the restricted execution so far.

As far as you know.  Every change is a potential security hole.

> One idea I was wondering is to provide a "hack me" machine somewhere on
> the web, including a hall-of-fame for those who break the system
> somehow. Hummm.. I must find some infrastructure to do that. Anyway,
> that could help us find some obvious problems quickly, but I understand
> it shouldn't affect the general decision of how to support restricted
> execution in Python.

Better put it in a serious chroot jail.  I think this has been tried
and broken into before.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From niemeyer@conectiva.com  Mon Dec 16 22:03:10 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Mon, 16 Dec 2002 20:03:10 -0200
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <200212162137.gBGLbkl07474@odiug.zope.com>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <200212162137.gBGLbkl07474@odiug.zope.com>
Message-ID: <20021216200309.B10940@ibook.distro.conectiva>

> > Perhaps the main features could be left in the core, without any real
> > application being offered as part of the "batteries". This way, a secure
> > environment could evolve in parallel, without forking the Python
> > development itself.
> 
> Of course that assumes that the main features in the core are secure.
> Samuele's observation that restricted code can modify a new-style
> class passed in belies that.

How that would affect Python itself, if no rexec module was offered? He
wouldn't even notice that this problem was there, unless he was using
the "python-rexec" external module. Even then, he would have reported
this as a bug in "python-rexec", not in Python itself.

> > I was talking about major changes like type/class unification. I belive
> > that major changes like this won't happen often, and that's the kind of
> > change that affected the restricted execution so far.
> 
> As far as you know.  Every change is a potential security hole.

I meant that the problems I have seen so far were caused by major
changes. I agree that it's hard to predict what change is going to
break it.

> Better put it in a serious chroot jail.

Indeed. It'd be good if that machine was completely dedicated to this
purpose.

> I think this has been tried and broken into before.

That's the goal! To get into the hall-of-fame, one would have to tell
how he got in. We can also send an email to someone just before
executing the code, so that even anonymous hackers contribute with the
idea.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From jeremy@alum.mit.edu  Mon Dec 16 22:06:29 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 16 Dec 2002 17:06:29 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <20021216190202.A11682@ibook.distro.conectiva>
References: <015b01c2a385$97010700$6d94fea9@newmexico>
 <200212161631.gBGGVFB01638@odiug.zope.com>
 <20021216171936.A10545@ibook.distro.conectiva>
 <200212161947.gBGJlYA05956@odiug.zope.com>
 <20021216190202.A11682@ibook.distro.conectiva>
Message-ID: <15870.20069.298805.666542@slothrop.zope.com>

>>>>> "GN" == Gustavo Niemeyer <niemeyer@conectiva.com> writes:

  GN> One idea I was wondering is to provide a "hack me" machine
  GN> somewhere on the web, including a hall-of-fame for those who
  GN> break the system somehow. Hummm.. I must find some
  GN> infrastructure to do that. Anyway, that could help us find some
  GN> obvious problems quickly, but I understand it shouldn't affect
  GN> the general decision of how to support restricted execution in
  GN> Python.

Unless there's some reason to believe this would be hard, what's the
point?

Jeremy



From pobrien@orbtech.com  Mon Dec 16 22:09:03 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Mon, 16 Dec 2002 16:09:03 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
Message-ID: <200212161609.03192.pobrien@orbtech.com>

--------------Boundary-00=_37G8NHCY5EHYZUUF98V5
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 8bit

I'd like to call attention to a bug report I just submitted to SF 
concerning a potentially serious problem with the c-version of pickle.

In short, I think I discovered a situation where cPickle is not doing 
the right thing in terms of storing a reference to a previously pickled 
class instance (using Pickler, of course). It looks like its doing a 
put of a new instance, then a get of the original instances dictionary, 
rather than a get of the original instance. On the other hand, pickle 
does the right thing.

I think this is a pretty serious problem, but I'm out of my league when 
it comes to debugging the c-version of pickle. I'm also basing my 
assessment on a minimal understanding of the pickle source code 
combined with looking at two pickle files, one created using pickle, 
the other using cPickle, with no other changes to the program that 
produced them. I understand that cosmetic differences are to be 
expected. But the difference I see is more substantial.

I attached the two pickle files to the SF bug report, but I can't tell 
if they got through so I'm attaching them here as well. If anyone can 
help track this down, I'd really appreciate it. If more detail is 
needed, or a simple code example would help, I'll be happy to do that.

Thanks.

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------

--------------Boundary-00=_37G8NHCY5EHYZUUF98V5
Content-Type: text/plain;
  charset="us-ascii";
  name="cpickle.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cpickle.txt"

(ccopy_reg
_reconstructor
p1
(cpypersyst.clock
Clock
p2
c__builtin__
object
p3
NtRp4
(dp5
S'_ispaused'
p6
I1
sS'_time'
p7
F1039910904.277091
sb(i__main__
Bank
p8
(dp9
S'_nextAccountNumber'
p10
I3
sS'_accounts'
p11
(dp12
I1
(iaccount
Account
(dp13
S'balance'
p14
I0
sS'number'
p15
I1
sbsI2
(iaccount
Account
(dp16
g14
I0
sg15
I2
sbssbtp17
.g1
(ccommands
DepositOrWithdrawal
p18
g3
NtRp19
(dp20
g7
F1039910911.252528
sS'acnt'
p21
(iaccount
Account
p22
g13
bsS'amount'
p23
I555
sb.g1
(g18
g3
NtRp24
(dp25
g7
F1039910918.068189
sg21
g22
sg23
I555
sb.
--------------Boundary-00=_37G8NHCY5EHYZUUF98V5
Content-Type: text/plain;
  charset="us-ascii";
  name="pickle.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="pickle.txt"

(ccopy_reg
_reconstructor
p0
(cpypersyst.clock
Clock
p1
c__builtin__
object
p2
Ntp3
R(dp4
S'_ispaused'
p5
I1
sS'_time'
p6
F1039911604.515402
sbp7
(i__main__
Bank
p8
(dp9
S'_nextAccountNumber'
p10
I3
sS'_accounts'
p11
(dp12
I1
(iaccount
Account
p13
(dp14
S'balance'
p15
I0
sS'number'
p16
I1
sbsI2
(iaccount
Account
p17
(dp18
g15
I0
sg16
I2
sbssbtp19
.g0
(ccommands
DepositOrWithdrawal
p20
g2
Ntp21
R(dp22
g6
F1039911613.244611
sS'acnt'
p23
g13
sS'amount'
p24
I555
sbp25
.g0
(g20
g2
Ntp26
R(dp27
g6
F1039911618.384868
sg23
g13
sg24
I555
sbp28
.
--------------Boundary-00=_37G8NHCY5EHYZUUF98V5--



From guido@python.org  Mon Dec 16 22:14:49 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 17:14:49 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: Your message of "Mon, 16 Dec 2002 20:03:10 -0200."
 <20021216200309.B10940@ibook.distro.conectiva>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <200212162137.gBGLbkl07474@odiug.zope.com>
 <20021216200309.B10940@ibook.distro.conectiva>
Message-ID: <200212162214.gBGMEnn10102@odiug.zope.com>

> > Of course that assumes that the main features in the core are secure.
> > Samuele's observation that restricted code can modify a new-style
> > class passed in belies that.
> 
> How that would affect Python itself, if no rexec module was offered? He
> wouldn't even notice that this problem was there, unless he was using
> the "python-rexec" external module. Even then, he would have reported
> this as a bug in "python-rexec", not in Python itself.

I was assuming that even if you're not using the rexec module, you're
using *some* way to execute untrusted Python code.  If that untrusted
Python code is given access to a new-style class, the hole is as open
as ever.  (Fortunately you can't modify built-in types, otherwise the
hole would be much bigger and much harder to avoid.)

> That's the goal! To get into the hall-of-fame, one would have to tell
> how he got in. We can also send an email to someone just before
> executing the code, so that even anonymous hackers contribute with the
> idea.

I don't expect you'll ever reach the point where it'll be wise to
advertise this as safe.  I certainly won't.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From niemeyer@conectiva.com  Mon Dec 16 22:18:14 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Mon, 16 Dec 2002 20:18:14 -0200
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <15870.20069.298805.666542@slothrop.zope.com>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <15870.20069.298805.666542@slothrop.zope.com>
Message-ID: <20021216201814.C10940@ibook.distro.conectiva>

> Unless there's some reason to believe this would be hard, what's the
> point?

The point is to catch the security flaws, and fix them.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From bac@OCF.Berkeley.EDU  Mon Dec 16 22:28:02 2002
From: bac@OCF.Berkeley.EDU (Brett Cannon)
Date: Mon, 16 Dec 2002 14:28:02 -0800 (PST)
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <200212161631.gBGGVFB01638@odiug.zope.com>
References: <015b01c2a385$97010700$6d94fea9@newmexico>
 <200212161631.gBGGVFB01638@odiug.zope.com>
Message-ID: <Pine.SOL.4.51.0212161422420.9499@death.OCF.Berkeley.EDU>

[Guido van Rossum]

> using this?  I suppose a fix would be simple enough, but I'm more and
> more inclined to simply rip out rexec from the distribution -- it's
> never going to be safe, and I doubt it's very useful as long as it's
> not safe.
>

Obviously Guido wants to just ditch rexec with the expectation it will
never work well enough for it to be legitimately called "safe".  Gustavo,
on the other hand, seems rather determined to get the thing to work.  Can
we just move it out of circulation to the nondist/ directory?  It seems
Gustavo does not want to move it to a completely separate project (maybe I
am misreading his emails, though).

But if moving it there is not reasonable, then I am voting for completely
removing it.  The code will still be in older versions if someone decides
to pick it up and work on it as a separate project.  And perhaps it could
be started from scratch for Python 3 when unification is done and there
are less things to have to worry about.

-Brett


From jeremy@alum.mit.edu (Jeremy Hylton)  Mon Dec 16 22:24:29 2002
From: jeremy@alum.mit.edu (Jeremy Hylton) (Jeremy Hylton)
Date: Mon, 16 Dec 2002 17:24:29 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <20021216201814.C10940@ibook.distro.conectiva>
References: <015b01c2a385$97010700$6d94fea9@newmexico>
 <200212161631.gBGGVFB01638@odiug.zope.com>
 <20021216171936.A10545@ibook.distro.conectiva>
 <200212161947.gBGJlYA05956@odiug.zope.com>
 <20021216190202.A11682@ibook.distro.conectiva>
 <15870.20069.298805.666542@slothrop.zope.com>
 <20021216201814.C10940@ibook.distro.conectiva>
Message-ID: <15870.21149.166400.97302@slothrop.zope.com>

>>>>> "GN" == Gustavo Niemeyer <niemeyer@conectiva.com> writes:

  >> Unless there's some reason to believe this would be hard, what's
  >> the point?

  GN> The point is to catch the security flaws, and fix them.

If it isn't hard to hack or if there are a lot of flaws, there isn't
much incentive for someone to attack it.

Jeremy



From greg@cosc.canterbury.ac.nz  Mon Dec 16 22:28:26 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Tue, 17 Dec 2002 11:28:26 +1300 (NZDT)
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212161145.21295.mmclay@comcast.net>
Message-ID: <200212162228.gBGMSQu07742@oma.cosc.canterbury.ac.nz>

Michael McLay <mclay@nist.gov>:

> I could change the name from decimal() to FixedPoint(), but the
> capital letters in the name scream "ugly".

Worse, it would completely lose the connotation that it is a *decimal*
number type, which surely is the main point of it in the first place!

If it's somehow important that the name contain at least one capital
letter, how about Decimal, or perhaps FixedDecimal if it's felt
necessary to distinguish it from some potential floating decimal type.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From guido@python.org  Mon Dec 16 22:34:56 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Dec 2002 17:34:56 -0500
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: Your message of "Mon, 16 Dec 2002 20:18:14 -0200."
 <20021216201814.C10940@ibook.distro.conectiva>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <15870.20069.298805.666542@slothrop.zope.com>
 <20021216201814.C10940@ibook.distro.conectiva>
Message-ID: <200212162234.gBGMYuq10231@odiug.zope.com>

> > Unless there's some reason to believe this would be hard, what's the
> > point?
> 
> The point is to catch the security flaws, and fix them.

I think that Jeremy meant that this is only a useful occupation if you
expect to eventually reach a point where you expect that there aren't
any security flaws left.  Jeremy & I both doubt that Python will ever
reach that level, meaning that the whole exercise of fixing security
flaws is a waste of time (if you know you *can't* make it safe, don't
waste time trying).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From lists@morpheus.demon.co.uk  Mon Dec 16 22:54:44 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Mon, 16 Dec 2002 22:54:44 +0000
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path
 manipulation (was Re: directory hierarchy proposal)
References: <r01050400-1022-BD2FA884112A11D7AD06003065D5E7E4@[10.0.0.23]> <200212161941.gBGJftc05924@odiug.zope.com>
Message-ID: <n2m-g.8yyppoaz.fsf@morpheus.demon.co.uk>

Guido van Rossum <guido@python.org> writes:

>> Playing tricks with the contents of pkg.__path__ apparently has its
>> uses,
>
> It's the *defined* API for adding to the set of locations from which
> submodules and subpackages of a package are to be loaded.  You can't
> just call that "playing tricks."

This is entirely true. I think the key issue for import hooks is
whether application code is entitled to make assumptions about what
the system puts onto pkg.__path__ before __init__.py gets control.

But the writer of __init__.py knows how the package is being packaged,
and hence what import hook is responsible for setting up __path__. So
it can be coded appropriately.

The mistake here is in assuming that arbitrary code can be
transparently zipped (or stored in another way, using an import hook)
and still work. Some (most) code can, but not all.

> Of course.  If you freeze code, you have to make sure that it will be
> able to find its data files in some other way.  But not all code has
> to be freezable.

This agrees with what I just said (with "freeze" replaced by "zip" or
whatever).

Paul
-- 
This signature intentionally left blank


From just@letterror.com  Mon Dec 16 23:29:53 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 17 Dec 2002 00:29:53 +0100
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: <200212161941.gBGJftc05924@odiug.zope.com>
Message-ID: <r01050400-1022-4EADB3E6114E11D7AD06003065D5E7E4@[10.0.0.23]>

Guido van Rossum wrote:

> > Playing tricks with the contents of pkg.__path__ apparently has its
> > uses,
> 
> It's the *defined* API for adding to the set of locations from which
> submodules and subpackages of a package are to be loaded.  You can't
> just call that "playing tricks."

pkg.__path__ is a package-local equivalent of sys.path. But if a package
didn't *originate* from sys.path, it makes little sense to demand
pkg.__path__ be a list of strings.

But: it could then just as well be an empty list. With my patch I think
frozen package imports could even be fixed to do that. Hm, that would be
real easy in fact. Will try this out.

> Does the metahook also apply to submodules (or subpackage) of
> packages?

Of course it does. An importer must be able to handle submodules of
packages it loaded, if it wants. Frozen import works that way: in fact
you can see the frozen import mechanism as an imaginary importer on
sys.meta_path. sys.meta_path is simply traversed before
sys.path/pkg.__path__, always.

A big difference between my hooks and the classic
imp.find_module/load_module functions is that my hooks *always* get
called with the fully qualified module name (and have to look up
sys.path or pkg.__path__ by themselves if they need it), and
imp.find_module must be called with a (sub)name and a path argument. The
latter is fine for of sys.path-based imports but is a pain for any
other, as then you somehow need to reconstruct the parent name (which
actually led to freeze (ab)using __path__ to communicate the parent
name).

> I'd expect not.  (I haven't had the time to review your latest
> patches yet, in part due to this thread. :-)

Oh, this aspect hasn't changed since since I first posted it on sf, but
I think you spent more time in zipimport.c than in import.c ;-)

> Surely you shouldn't be looking for builtin submodules of a package.

Why not? (Builtin submodules won't work anyway at the moment, but for
other reasons.)

> >     Depending on the kind of importer, there are different
> >     levels of freedom of what you can use as pkg.__path__.
> >     
> >     Importer object on sys.meta_path:
> >         it can use anything it pleases (even None), as long
> >         as a __path__ variable is set.
> 
> But I can imagine that some metahooks would like to look inside the
> __path__ list for more hints on where to find the submodules of the
> package.  Of course that's up to the metahook.

Exactly. __path__ can be seen as private to an importer. It just happens
to be a list of strings for the sys.path importer ;-).

> >     Importer object on sys.path:
> >         pkg.__path__ must be a list; it's most logical to use
> >         an importer object as the only item. Could be the same
> >         importer instance that imported the package itself.
> 
> Multiple items would make sense too.

Sure.

> >     A hook on sys.path_hooks:
> >         pkg.__path__ must be a list and its only item should
> >         be a string that the hook can handle itself.
> 
> IMO it should allow multiple strings too!

It sure does, I formulated that clumsily. If no meta_path hook picks up
the import, sys.path or pkg.__path__ traversal does its work as usual.
Apart from the sys.path_hooks hookability of sys.path/pkg.__path__
items, nothing was changed here.

> >     These are just guidelines: a set of hooks could in theory
> >     deliberately set pgk.__path__ up so submodule imports be
> >     handled by an entirely different importer. Not sure how
> >     useful that would be...
> 
> Very useful, for not-yet-imagined cases.

I was just pointing out that it's possible...

Just


From skip@pobox.com  Mon Dec 16 18:25:28 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 16 Dec 2002 12:25:28 -0600
Subject: [Python-Dev] summary for 2002-12-01 through 2002-12-15
In-Reply-To: <Pine.SOL.4.51.0212152302580.6942@death.OCF.Berkeley.EDU>
References: <Pine.SOL.4.51.0212152302580.6942@death.OCF.Berkeley.EDU>
Message-ID: <15870.6808.471292.407695@montanaro.dyndns.org>

    Brett> =========================================================
    Brett> `extension module .o files wind up in the wrong place`__
    Brett> =========================================================
    Brett> __ http://mail.python.org/pipermail/python-dev/2002-December/030644.html

    Brett> Skip Montanaro discovered that if you built Python in a subdirectory of
    Brett> the source directory (e.g., source in ``source/`` and you are building in
    Brett> ``source/build/``) the .o files are put in the wrong place (they use
    Brett> ``source/`` as the root directory to place things instead of
    Brett> ``source/build/``).  Skip resolved his issue by doing a ``make distclean``
    Brett> and then running ``../configure``.

Actually, no, I haven't yet resolved this.  Someone else said 'make
distclean' seemed to work for them.  For the time being I'm just using two
CVS checkouts and building in the source tree.  I need to see if a bug
report needs filing...

Skip


From neal@metaslash.com  Tue Dec 17 01:49:35 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Mon, 16 Dec 2002 20:49:35 -0500
Subject: [Python-Dev] deprecating APIs
Message-ID: <20021217014935.GQ16807@epoch.metaslash.com>

Are we still planning to provide a mechanism to declare APIs, etc as
deprecated?

I have this code added to pyport.h which works for gcc:

    /* Py_DEPRECATED(version)
     * Declare a macro or function deprecated.
     * Usage:
     *    extern int old_var Py_DEPRECATED(2.3);
     *    typedef int T1 Py_DEPRECATED(2.4);
     *    extern int x() Py_DEPRECATED(2.5);
     */
    #if defined(__GNUC__) && (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1)
    #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
    #else
    #define Py_DEPRECATED(VERSION_UNUSED) 
    #endif

The version number is not necessary.  It is only for documentation.
I don't know how to support other compilers.

We could add Py_USE_DEPRECATED and/or Py_HIDE_DEPRECATED for
providing/hiding access to deprecated features.

    #ifdef Py_USE_DEPRECATED
    #define Py_SOME_DEPRECATED_MACRO
    #endif

or

    #ifndef Py_HIDE_DEPRECATED
    #define Py_SOME_DEPRECATED_MACRO
    #endif

Unfortunately, the Py_DEPRECATED macro doesn't work with macros, so we
would have to use Py_USE_DEPRECATED/Py_HIDE_DEPRECATED around
deprecated macros.

Neal



From tim.one@comcast.net  Tue Dec 17 05:34:35 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Dec 2002 00:34:35 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212161145.21295.mmclay@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEOADCAB.tim.one@comcast.net>

[Michael McLay]
> Fixing the decimal place would make sense for applications having
> to do with money. That is quite different than how numbers are
> treated in scientific calculations.

Of course!  FixedPoint was written for people with fixed-point applications,
and especially for those of such who think a computer should be as good at
decimal math as their 12-year old daughter driving a $1 hand calculator
<0.500021 wink>.

> That is were I borrowed the idea of significant digits.  Accountants
> tend to think that all digits are significant, down to the last
> penny.

"The last penny" is an inherently fixed-point concept:  the last penny is
measured in units of 0.01 regardless of how many digits appear before the
decimal point.  That's what FixedPoint models directly.  The total number of
digits is unbounded, much like the national debt, but the position of the
decimal point is fixed.

> ...
> My goal is to have a new numerical type added to Python that
> satisfies the requirements for accountants and newbies.  Your
> description of fixed point seems to meet this goal.

That was its intent, but it is an expensive flavor of arithmetic, and not to
everyone's tastes.

> My confusion was raised by the 'e' notation seeming to add precision
> to numbers (e.g. 1.0e5 expands to 100000.0.)

Well, that didn't come from the FixedPoint module -- it always gives a
number of digits after the decimal point equal to the number the user tells
it to use; it doesn't try to infer anything from the form of a literal.

> This would be incorrect for reporting significant digits in scientific
> papers, but the rules for accounting math are different. This was
> simply my confusion.  FixedPoint appears to be sufficient for bankers
> and accountants.

I believe so, but IANAB and IANAA.

> ...
> This is a very good long term goal.  So for Python3k the standard
> numerical type will be Cowlishaw's decimal numbers.

Don't know.  It brings its own kinds of problems, not least of which is
sloth.

> To bad the decNumber library from IBM is not open source.

A Java implementation is open:

    http://www2.hursley.ibm.com/decimalj/

> In the short term what do you advise be done with the patch to
> add a fixed point literal and a builtin fixed point conversion
> function?

In the short term it's most important to get FixedPoint into 2.3 (under
whatever name people like).  One thing at a time is more than I can handle
these days.

> I could change the name from decimal() to FixedPoint(), but the
> capital letters in the name scream "ugly".

I don't care about the name.

> This isn't consistent with the short names used by
> the other numerical conversion functions (int, float, hex, oct...).
> Would fixed() be an acceptable name?

Make everyone unhappy and call it decimal_fixed_point() <0.9 wink>.  I'm not
sure why you might prefer fixed() to decimal(), BTW.  One reason I would is
that decimal() reminds of oct() and hex(), but would do something very
different.

> This also suggests the suffix should be changed to 'f' and
> 'F' instead of 'd' and 'D'.

-1.  'f' is a legit suffix for a float literal in C, and so much of Python's
literal syntax is identical to C's that reusing part of it to mean something
else seems a Bad Idea.

>>     http://www2.hursley.ibm.com/decimal/decifaq.html

> According to this reference the most frequent numerical column
> type used in commercial databases is decimal (55%). The float type
> is used just 1.4% of the time with the rest being split between
> Integer and SmallInt.
> ...
> Is it safe to conclude that Python will be be much more appealing
> to a huge segment of the IT market if grows a native decimal number
> type?

I've never worked in the IT market and don't know what drives them.  From
email received in response to FixedPoint, all I've gleaned is that (1) they
want arithmetic that exactly mimics whatever idiosyncratic schemes are used
by a huge variety of software systems I never heard of; and, (2) they're
unable to explain what those schemes are well enough to mimic them.  Most
people in this business have no idea what "the rules" are, they just know
that their butt is on the line if they deviate from what The Company has
been seeing for the last 20 years.  So, e.g., you get requests for flatly
insane rounding rules, like "if the fraction is over 0.4, round away from 0,
else round toward it".  Most of the time it turns out that isn't really the
rule they need, either, it's just the first one they generalized from one
example, after failing to find docs.

FixedPoint is extremely careful to give the best possible numeric result in
all cases.  I do know that isn't really what the IT market cares about --
compatability with legacy systems-- themselves incompatible with each
other --is much more important to them.  OTOH, giving the best possible
result is, as the IEEE-754 fp standard demonstrated with force in practice,
explainable, reproducible, and ultimately compelling.  That it's also
expensive is just icing on the cake <wink>.

[Michael McLay]
>> I could change the name from decimal() to FixedPoint(), but the
>> capital letters in the name scream "ugly".

[Greg Ewing]
> Worse, it would completely lose the connotation that it is a *decimal*
> number type, which surely is the main point of it in the first place!

Only to a compsci type.  The intended audience never heard of binary
fixed-point -- fixed-point means decimal to them as surely as New Zealand
means anarchy to an American <wink>.

> If it's somehow important that the name contain at least one capital
> letter,

It isn't.



From martin@v.loewis.de  Tue Dec 17 07:12:52 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 17 Dec 2002 08:12:52 +0100
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212161609.03192.pobrien@orbtech.com>
References: <200212161609.03192.pobrien@orbtech.com>
Message-ID: <m3vg1tqft7.fsf@mira.informatik.hu-berlin.de>

"Patrick K. O'Brien" <pobrien@orbtech.com> writes:

> I'd like to call attention to a bug report I just submitted to SF 
> concerning a potentially serious problem with the c-version of pickle.
[...]
> I think this is a pretty serious problem, but I'm out of my league when 
> it comes to debugging the c-version of pickle. 

I hear such statements quite often recently, but I'm not impressed.
You have the full source, so there is no principle problem that would
make it impossible to investigate the problem yourself. It might take
you more time to investigate this than it may take somebody else, but
I'm sure you have heard about the printf function; if not, you should
read some book first.

Understanding software that does not work right is *always* difficult,
and time-consuming, because it usually looks right - atleast that's
what the author of the software thought when he wrote it.

It might be that you may not have the time to investigate this, but
then, that might hold for everybody else.

Regards,
Martin


From mmclay@comcast.net  Tue Dec 17 11:39:48 2002
From: mmclay@comcast.net (Michael McLay)
Date: Tue, 17 Dec 2002 06:39:48 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEOADCAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCEOADCAB.tim.one@comcast.net>
Message-ID: <200212170639.33487.mclay@nist.gov>

On Tuesday 17 December 2002 12:34 am, Tim Peters wrote:
> [Michael McLay]
>
> > Fixing the decimal place would make sense for applications having
> > to do with money. That is quite different than how numbers are
> > treated in scientific calculations.
>
> Of course!  FixedPoint was written for people with fixed-point
> applications, and especially for those of such who think a computer should
> be as good at decimal math as their 12-year old daughter driving a $1 hand
> calculator <0.500021 wink>.
>
> > That is were I borrowed the idea of significant digits.  Accountants
> > tend to think that all digits are significant, down to the last
> > penny.
>
> "The last penny" is an inherently fixed-point concept:  the last penny is
> measured in units of 0.01 regardless of how many digits appear before the
> decimal point.  That's what FixedPoint models directly.  The total number
> of digits is unbounded, much like the national debt, but the position of
> the decimal point is fixed.

It's always good to plan ahead for the government's budget. Thanks. 

> > My goal is to have a new numerical type added to Python that
> > satisfies the requirements for accountants and newbies.  Your
> > description of fixed point seems to meet this goal.
>
> That was its intent, but it is an expensive flavor of arithmetic, and not
> to everyone's tastes.

Yes, but since it does what most people expect, it will make Python more 
appealing to the average Joe. Now that I have a better understanding of 
Cowlishaw's decimal numbers I can see how they would be a better solution, 
but for the short term having numbers that behave like most calculators would 
be a reasonable start. I am concerned about one thing. Will the fixed point 
implementation you have implemented be backwards compatible with an 
implementation of Cowlishaw's decimal numbers?

> > My confusion was raised by the 'e' notation seeming to add precision
> > to numbers (e.g. 1.0e5 expands to 100000.0.)
>
> Well, that didn't come from the FixedPoint module -- it always gives a
> number of digits after the decimal point equal to the number the user tells
> it to use; it doesn't try to infer anything from the form of a literal.

Most of the time this is probably the right thing to do. Bit I think it might 
be a good idea to infer the precision from string literals if no precision is 
explicitly defined. As it is currently implemented the behavior is:

> fixed("12.3")
12.30

The addition of '0' is unexpected. I can see the need to make specify 
precision when the value being converted is a floating point number, but if 
it is a string then the number is a literal value specified by the user. They 
should get exactly what they write. That would be:

> fixed("12.3")
fixed(12.3,1)
> fixed("12.305")
fixed(12.305,3)

Maybe this is too subtle of a rule, but without this rule it will be difficult 
to implement a literal that doesn't require the explicit definition of a 
level of precision. So, looking forward the rule for adding precision should 
be a default to 2 decimal places for floats and integers, but it should be a 
literal interpretation of the value passed to the constructor as a string. 
For conversions for floats the results would be:

> fixed(12.3)
fixed(12.30,2)
> fixed(12.305)
fixed(12.30,2)
> fixed(12.3,3)
fixed(12.300,3)
> fixed(12.305,1)
fixed(12.3,1)
> fixed(12)
fixed(12.00,2)

(Perhaps it should be a default to a precision of 0 if the value passed is an 
integer.)

Ideally the proposal for adding literals would be added to 2.3, but Guido 
vetoed that, and the literals would derive the precision from the number of 
trailing zeros that were written:

> 12.3d
12.3
> 12.300d
12.300

If the rule for defining the default precision doesn't make the exception for 
a literal string then it will be difficult to add a syntactic literal that 
doesn't require the explicit definition of the level of precision. This would 
end up with an awkward syntax, such as:

> 12.3:1d
12.3
> 12.30:3d
12.300

> > This would be incorrect for reporting significant digits in scientific
> > papers, but the rules for accounting math are different. This was
> > simply my confusion.  FixedPoint appears to be sufficient for bankers
> > and accountants.
>
> I believe so, but IANAB and IANAA.

And neither am I. But considering the inconsistencies in their current 
expectations I think you have defined a class that will have few objections.
> > ...
> > This is a very good long term goal.  So for Python3k the standard
> > numerical type will be Cowlishaw's decimal numbers.
>
> Don't know.  It brings its own kinds of problems, not least of which is
> sloth.

Has someone accused you of sloth? Hardly. I'm sure I'll irritate Guido by 
suggesting the addition of a second parser to Python, but through this 
mechanism I think it will be possible to make a natural and relatively 
painless transition to using Cowlishaw's numbers as the default number type 
in Python 3k 

> > To bad the decNumber library from IBM is not open source.
>
> A Java implementation is open:
>
>     http://www2.hursley.ibm.com/decimalj/

That's great for jython, but it will require more work for the C Python 
implementation.

> > In the short term what do you advise be done with the patch to
> > add a fixed point literal and a builtin fixed point conversion
> > function?
>
> In the short term it's most important to get FixedPoint into 2.3 (under
> whatever name people like).  One thing at a time is more than I can handle
> these days.

Typing "FixedPoint" instead of "fixed" will be quite irritating, but I'll be 
happy with either name if the class is included in the standard distribution.

> > I could change the name from decimal() to FixedPoint(), but the
> > capital letters in the name scream "ugly".
>
> I don't care about the name.

I'd rather have a literal added to the language and a builtin "fixed" as the 
name, but Guido has vetoed the literal. Again. I'll be happy with anything 
that adds the capabilities to the standard distribution.

> > This isn't consistent with the short names used by
> > the other numerical conversion functions (int, float, hex, oct...).
> > Would fixed() be an acceptable name?
>
> Make everyone unhappy and call it decimal_fixed_point() <0.9 wink>.  I'm
> not sure why you might prefer fixed() to decimal(), BTW.  One reason I
> would is that decimal() reminds of oct() and hex(), but would do something
> very different.

Good idea, calling it decimal_fixed_point should maximize the irritation of 
end users. This will lead to more pressure to add a literal to the language.

I thought it might be best to use "fixed" for fixed point in case there was 
some incompatibility with the eventual inclusion of Cowlishaw's decimal 
numbers. The intention was to reserve "decimal" for the more generalized form 
of decimal numbers. If your FixedPoint implementation is 100% compatible then 
using "decimal" would be best. Being able to extend FixedPoint semantics to 
include floating point decimals in the future would be the best option. 

> > This also suggests the suffix should be changed to 'f' and
> > 'F' instead of 'd' and 'D'.
>
> -1.  'f' is a legit suffix for a float literal in C, and so much of
> Python's literal syntax is identical to C's that reusing part of it to mean
> something else seems a Bad Idea.

I didn't like the change for the same reason, but I made the change so that 
'd' and 'D' could be reseved for Cowlishaw's decimal numbers. If they are 
compatible then I withdraw the change from my original proposal.

> >>     http://www2.hursley.ibm.com/decimal/decifaq.html
> >
> > According to this reference the most frequent numerical column
> > type used in commercial databases is decimal (55%). The float type
> > is used just 1.4% of the time with the rest being split between
> > Integer and SmallInt.
> > ...
> > Is it safe to conclude that Python will be be much more appealing
> > to a huge segment of the IT market if grows a native decimal number
> > type?
>
> I've never worked in the IT market and don't know what drives them.  From
> email received in response to FixedPoint, all I've gleaned is that (1) they
> want arithmetic that exactly mimics whatever idiosyncratic schemes are used
> by a huge variety of software systems I never heard of; and, (2) they're
> unable to explain what those schemes are well enough to mimic them.  Most
> people in this business have no idea what "the rules" are, they just know
> that their butt is on the line if they deviate from what The Company has
> been seeing for the last 20 years.  So, e.g., you get requests for flatly
> insane rounding rules, like "if the fraction is over 0.4, round away from
> 0, else round toward it".  Most of the time it turns out that isn't really
> the rule they need, either, it's just the first one they generalized from
> one example, after failing to find docs.
>
> FixedPoint is extremely careful to give the best possible numeric result in
> all cases.  I do know that isn't really what the IT market cares about --
> compatability with legacy systems-- themselves incompatible with each
> other --is much more important to them.  OTOH, giving the best possible
> result is, as the IEEE-754 fp standard demonstrated with force in practice,
> explainable, reproducible, and ultimately compelling.  That it's also
> expensive is just icing on the cake <wink>.

I think you have done a fine job of providing what they need, regardless of 
what the end users think they want. It will be easy enough to provide 
conversion capabilities to the broken semantics of the sundry implementation 
of decimal numbers that exist.

> [Michael McLay]
>
> >> I could change the name from decimal() to FixedPoint(), but the
> >> capital letters in the name scream "ugly".
>
> [Greg Ewing]
>
> > Worse, it would completely lose the connotation that it is a *decimal*
> > number type, which surely is the main point of it in the first place!
>
> Only to a compsci type.  The intended audience never heard of binary
> fixed-point -- fixed-point means decimal to them as surely as New Zealand
> means anarchy to an American <wink>.

Let's go easy on the kiwis. They don't always get their due. It looks like 
"Mad" Pearse [1] may have flown before the Wright bothers, but since he was a 
perfectionist he did not try to claim the title. <wink>
 
[1] http://slashdot.org/article.pl?sid=02/12/16/0338254&mode=thread&tid=126



From mwh@python.net  Tue Dec 17 12:39:46 2002
From: mwh@python.net (Michael Hudson)
Date: 17 Dec 2002 12:39:46 +0000
Subject: [Python-Dev] summary for 2002-12-01 through 2002-12-15
In-Reply-To: Skip Montanaro's message of "Mon, 16 Dec 2002 12:25:28 -0600"
References: <Pine.SOL.4.51.0212152302580.6942@death.OCF.Berkeley.EDU> <15870.6808.471292.407695@montanaro.dyndns.org>
Message-ID: <2my96obyzx.fsf@starship.python.net>

Skip Montanaro <skip@pobox.com> writes:

> Actually, no, I haven't yet resolved this.  Someone else said 'make
> distclean' seemed to work for them.  For the time being I'm just using two
> CVS checkouts and building in the source tree.  I need to see if a bug
> report needs filing...

I reopened the bug (#599248) whose fix caused the problem.  Last news
was that AMK was "thinking about it".  I guess we could just back it
out until such time as someone comes up with a better fix.

Cheers,
M.

-- 
  US elections
  For those of you fearing that the rest of the world might be 
  making fun of the US because of this: Rest assured, we are.
         -- http://www.advogato.org/person/jameson/diary.html?start=12


From pobrien@orbtech.com  Tue Dec 17 15:12:11 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Tue, 17 Dec 2002 09:12:11 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <m3vg1tqft7.fsf@mira.informatik.hu-berlin.de>
References: <200212161609.03192.pobrien@orbtech.com> <m3vg1tqft7.fsf@mira.informatik.hu-berlin.de>
Message-ID: <200212170912.11638.pobrien@orbtech.com>

On Tuesday 17 December 2002 01:12 am, Martin v. Löwis wrote:
> "Patrick K. O'Brien" <pobrien@orbtech.com> writes:
> > I'd like to call attention to a bug report I just submitted to SF
> > concerning a potentially serious problem with the c-version of
> > pickle.
>
> [...]
>
> > I think this is a pretty serious problem, but I'm out of my league
> > when it comes to debugging the c-version of pickle.
>
> I hear such statements quite often recently, but I'm not impressed.

You don't hear statements like that from me. And impressing people on 
this list wasn't my goal. So I'm at a complete loss as to what I've 
done to deserve this kind of response from you.

> You have the full source, so there is no principle problem that would
> make it impossible to investigate the problem yourself. It might take
> you more time to investigate this than it may take somebody else, but
> I'm sure you have heard about the printf function; if not, you should
> read some book first.

And you don't think my asking for some help on this list is better use 
of two people's time? One who has an example that fails, and one who 
knows the cPickle implementation well enough to help track down the 
cause of failure.

I've already spent quite a lot of time tracking the problem down to the 
point that I have. And it is trivial for me to fix my own code so that 
the problem goes away. But the underlying problem isn't with my code. 
It is with cPickle. And I'm concerned about just walking away from this 
because I think it is a serious problem for anyone who wants to 
correctly unpickle objects that were pickled using cPickle.

Fortunately, Guido has responded to my SF bug report and I'm trying to 
reduce things to a simple example that illustrates the failure. I spent 
several hours on this last night, and I'll likely spend several hours 
on it today. And that's just getting us to a simple example that 
reproduces the problem. Frankly, I don't have the time nor the interest 
to get back up to speed on C. (I've read lots of C books and dabbled in 
it, but that was years ago.)

> Understanding software that does not work right is *always*
> difficult, and time-consuming, because it usually looks right -
> atleast that's what the author of the software thought when he wrote
> it.

Sure. And this is the first time I've seen cPickle do the wrong thing. 
Considering that the app that found this bug is an example of a banking 
app that logs account transactions as pickles, I think you might 
appreciate that it would be a good thing if those pickles maintained 
proper references to those accounts and could restore transactions 
appropriately, rather than creating new account instances that weren't 
part of the bank. (Say goodbye to your money on a system recovery.)

> It might be that you may not have the time to investigate this, but
> then, that might hold for everybody else.

And what harm did I do by asking if anyone had time to help, rather than 
letting this issue drop? Sorry if this isn't the kind of response you 
expected back from me. But I'm already irritated with the amount of 
time I've spent on this. Your reply really rubbed me the wrong way. If 
I didn't care about Python I sure as hell wouldn't be spending my time 
tracking this problem down.

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------



From martin@v.loewis.de  Tue Dec 17 16:28:15 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 17 Dec 2002 17:28:15 +0100
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212170912.11638.pobrien@orbtech.com>
References: <200212161609.03192.pobrien@orbtech.com>
 <m3vg1tqft7.fsf@mira.informatik.hu-berlin.de>
 <200212170912.11638.pobrien@orbtech.com>
Message-ID: <m3znr4haow.fsf@mira.informatik.hu-berlin.de>

"Patrick K. O'Brien" <pobrien@orbtech.com> writes:

> > I hear such statements quite often recently, but I'm not impressed.
> 
> You don't hear statements like that from me. And impressing people on 
> this list wasn't my goal. So I'm at a complete loss as to what I've 
> done to deserve this kind of response from you.

Your original message sounded much like a support question. I was
trying to say (perhaps in impolite words, for which I apologize) that
posting support requests is not appropriate for this mailing list.

> And you don't think my asking for some help on this list is better use 
> of two people's time? One who has an example that fails, and one who 
> knows the cPickle implementation well enough to help track down the 
> cause of failure.

Valuing one person's time against the time of another person is always
a difficult issue - unless one person pays the other for the time.

> It is with cPickle. And I'm concerned about just walking away from this 
> because I think it is a serious problem for anyone who wants to 
> correctly unpickle objects that were pickled using cPickle.

And it is a good thing that you have submitted a bug report to
sourceforge.

> Sure. And this is the first time I've seen cPickle do the wrong thing. 
> Considering that the app that found this bug is an example of a banking 
> app that logs account transactions as pickles, I think you might 
> appreciate that it would be a good thing if those pickles maintained 
> proper references to those accounts and could restore transactions 
> appropriately, rather than creating new account instances that weren't 
> part of the bank.

I certainly agree that it is desirable to fix this bug, just like it
is desirable to fix all the other bugs.

> And what harm did I do by asking if anyone had time to help, rather than 
> letting this issue drop? Sorry if this isn't the kind of response you 
> expected back from me. But I'm already irritated with the amount of 
> time I've spent on this. Your reply really rubbed me the wrong way. If 
> I didn't care about Python I sure as hell wouldn't be spending my time 
> tracking this problem down.

It was the (perceived) attitude "my bug report is more important than
the other 320 open bug reports" that rubbed me the wrong way. I do
appreciate that you took the time to report the bug. 

This is all volunteer's time: users voluntarily take the time to
report bugs, Python developers voluntarily take the time to
investigate and fix them. The perceived seriousness of a problem is
but one factor for a contributor to determine whether to work on
the problem.

Regards,
Martin


From Wiktor Sadowski" <art@wiktorsadowski.com  Mon Dec 16 17:37:49 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Mon, 16 Dec 2002 18:37:49 +0100
Subject: [Python-Dev] Zipping Zope3
Message-ID: <005501c2a529$db51ab80$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_0052_01C2A532.3C12AD40
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

[Guido van Rossum]
>The problem is that Zope3 needs *some* way of storing data and config
>files related to specific packages, and there isn't really any other
>convenient and conventional way except storing it in a package
>directory and using the package's __file__ to find it.

__importer__ , the zip specific alternative to __file__ could be =
provided to solve the problem.

in an __init__.py:

if __file__:
    ............................
elif __importer__:
      f=3D___importer__.open()#to speedup and to get direct access to =
the binary
      #without  ___importer__.open()  should work as well                =
                  =20
      data1=3D___importer__.get_data("ModuleName\\index.html")
      data2=3D___importer__.get_data("ModuleName\\help.html")
      if data1:
      ..........
      f.close()

It would require some minor changes to Just's zipimport.c
+ new  function in import.c
PyImport_ExecCodeModuleEx2(char *name, PyObject *co, char =
*pathname,PyObject *importer)
where instead of setting __file__:
if (PyDict_SetItemString(d, "__importer__", importer) !=3D 0)
........................................................

As for zipimport.c:
new methods:
open() for speedup :
........
        char* fname=3DPyString_AsString(self->archive);
 self->arcfile=3DPyFile_FromString(fname,"rb");
 .......
 return self->arcfile; #PyObject*arcfile new field(zipimporter struct)
..................................................................
get_data(data_path)#modified load_module=20
..................................................................
in load_module:
mod =3D PyImport_ExecCodeModuleEx2(fullname, code, modpath,self);
instead of=20
mod =3D PyImport_ExecCodeModuleEx(fullname, code, modpath);

Tested;)

Regards
Wiktor
http://www.wiktorsadowski.com




------=_NextPart_000_0052_01C2A532.3C12AD40
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV>[Guido van Rossum]<BR>&gt;The problem is that Zope3 needs *some* =
way of=20
storing data and config<BR>&gt;files related to specific packages, and =
there=20
isn't really any other<BR>&gt;convenient and conventional way except =
storing it=20
in a package<BR>&gt;directory and using the package's __file__ to find =
it.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>__importer__ , the zip specific alternative to __file__ could=20
be&nbsp;provided to solve the problem.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>in an __init__.py:</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>if __file__:<BR>&nbsp;&nbsp;&nbsp; =
............................<BR>elif=20
__importer__:<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
f=3D___importer__.open()#to=20
speedup&nbsp;and to get direct access to the binary</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#without&nbsp;&nbsp;___importer_=
_.open()&nbsp;=20
should work as=20
well&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<BR>&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=20
data1=3D___importer__.get_data("ModuleName\\index.html")<BR>&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;=20
data2=3D___importer__.get_data("ModuleName\\help.html")<BR>&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;=20
if data1:<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=20
..........<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; f.close()</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>It would require some minor changes to Just's zipimport.c<BR>+ =
new&nbsp;=20
function in import.c<BR>PyImport_ExecCodeModuleEx2(char *name, PyObject =
*co,=20
char *pathname,PyObject *importer)<BR>where instead of setting =
__file__:<BR>if=20
(PyDict_SetItemString(d, "__importer__", importer) !=3D=20
0)<BR>........................................................</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>As for zipimport.c:<BR>new methods:<BR>open() for speedup=20
:<BR>........<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; char*=20
fname=3DPyString_AsString(self-&gt;archive);<BR>&nbsp;self-&gt;arcfile=3D=
PyFile_FromString(fname,"rb");<BR>&nbsp;.......<BR>&nbsp;return=20
self-&gt;arcfile; #PyObject*arcfile new field(zipimporter=20
struct)<BR>..............................................................=
....<BR>get_data(data_path)#modified=20
load_module=20
<BR>..................................................................<BR=
>in=20
load_module:<BR>mod =3D PyImport_ExecCodeModuleEx2(fullname, code,=20
modpath,self);<BR>instead of <BR>mod =3D =
PyImport_ExecCodeModuleEx(fullname, code,=20
modpath);</DIV>
<DIV>&nbsp;</DIV>
<DIV>Tested;)</DIV>
<DIV>&nbsp;</DIV>
<DIV>Regards<BR>Wiktor<BR><A=20
href=3D"http://www.wiktorsadowski.com">http://www.wiktorsadowski.com</A><=
/DIV>
<DIV>&nbsp;</DIV>
<DIV><BR>&nbsp;</DIV></BODY></HTML>

------=_NextPart_000_0052_01C2A532.3C12AD40--



From niemeyer@conectiva.com  Tue Dec 17 17:34:23 2002
From: niemeyer@conectiva.com (Gustavo Niemeyer)
Date: Tue, 17 Dec 2002 15:34:23 -0200
Subject: [Python-Dev] known obvious thing or bug (rexec)?
In-Reply-To: <200212162234.gBGMYuq10231@odiug.zope.com>
References: <015b01c2a385$97010700$6d94fea9@newmexico> <200212161631.gBGGVFB01638@odiug.zope.com> <20021216171936.A10545@ibook.distro.conectiva> <200212161947.gBGJlYA05956@odiug.zope.com> <20021216190202.A11682@ibook.distro.conectiva> <15870.20069.298805.666542@slothrop.zope.com> <20021216201814.C10940@ibook.distro.conectiva> <200212162234.gBGMYuq10231@odiug.zope.com>
Message-ID: <20021217153423.A17719@ibook.distro.conectiva>

> I think that Jeremy meant that this is only a useful occupation if you
> expect to eventually reach a point where you expect that there aren't
> any security flaws left.  Jeremy & I both doubt that Python will ever
> reach that level, meaning that the whole exercise of fixing security
> flaws is a waste of time (if you know you *can't* make it safe, don't
> waste time trying).

Ok.. I'll take your suggestion. Thank you.

-- 
Gustavo Niemeyer

[ 2AAC 7928 0FBF 0299 5EB5  60E2 2253 B29A 6664 3A0C ]


From just@letterror.com  Tue Dec 17 19:54:05 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 17 Dec 2002 20:54:05 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <005501c2a529$db51ab80$0201010a@wiktor>
Message-ID: <r01050400-1022-50C514A411F911D7AD06003065D5E7E4@[10.0.0.1]>

Wiktor Sadowski wrote:

> [Guido van Rossum]
> >The problem is that Zope3 needs *some* way of storing data and config
> >files related to specific packages, and there isn't really any other
> >convenient and conventional way except storing it in a package
> >directory and using the package's __file__ to find it.
> 
> __importer__ , the zip specific alternative to __file__ could be
> provided to solve the problem.

Hm, I like this!

> in an __init__.py:
> 
> if __file__:
>     ............................
> elif __importer__:
>       f=___importer__.open()#to speedup and to get direct access to
the binary
>       #without  ___importer__.open()  should work as well                                   
>       data1=___importer__.get_data("ModuleName\\index.html")
>       data2=___importer__.get_data("ModuleName\\help.html")
>       if data1:
>       ..........
>       f.close()

Or:

    try:
        __importer__
    except NameError:
        <deal with __file__>
    else:
        x = __importer__.get_data(...)

> It would require some minor changes to Just's zipimport.c
> + new  function in import.c
> PyImport_ExecCodeModuleEx2(char *name, PyObject *co, char
*pathname,PyObject *importer)
> where instead of setting __file__:
> if (PyDict_SetItemString(d, "__importer__", importer) != 0)
> .........................................................

I'd say it should set __importer__ in *addition* to __file__.

> As for zipimport.c:
> new methods:
> open() for speedup :
> .........
>         char* fname=PyString_AsString(self->archive);
>  self->arcfile=PyFile_FromString(fname,"rb");
>  .......
>  return self->arcfile; #PyObject*arcfile new field(zipimporter struct)
> ...................................................................
> get_data(data_path)#modified load_module 
> ...................................................................
> in load_module:
> mod = PyImport_ExecCodeModuleEx2(fullname, code, modpath,self);
> instead of 
> mod = PyImport_ExecCodeModuleEx(fullname, code, modpath);
> 
> Tested;)

Ugh, you should really consider getting a decent diff tool :-/ How can
*I* test if if I have no clue what exactly you've done?

I can see that keeping the file object open could speed things up, but
is it a polite & decent thing to do?

Just


From nas@python.ca  Tue Dec 17 20:14:27 2002
From: nas@python.ca (Neil Schemenauer)
Date: Tue, 17 Dec 2002 12:14:27 -0800
Subject: [Python-Dev] deprecating APIs
In-Reply-To: <20021217014935.GQ16807@epoch.metaslash.com>
References: <20021217014935.GQ16807@epoch.metaslash.com>
Message-ID: <20021217201427.GA2520@glacier.arctrix.com>

Neal Norwitz wrote:
> Are we still planning to provide a mechanism to declare APIs, etc as
> deprecated?

I think it would be a good idea.

  Neil


From Jack.Jansen@oratrix.com  Tue Dec 17 21:11:40 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Tue, 17 Dec 2002 22:11:40 +0100
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: <200212161747.gBGHlYK03859@odiug.zope.com>
Message-ID: <22D6ACBE-1204-11D7-BC1B-000A27B19B96@oratrix.com>

On maandag, dec 16, 2002, at 18:47 Europe/Amsterdam, Guido van Rossum 
wrote:
> - Nobody remembers why there is a special case in site.py for lines in
>   the .pth file that start with 'import', except that Martin von
>   Loewis once wanted this.  I propose not to duplicate this
>   functionality in .pkg files.

If you *really* *really* want to prepend an item to sys.path [Jack sits 
quietly to a thundering speech as to why you shouldn't want this, 
studying his fingernails and various other parts of his anatomy] then 
the "import" hack in .pth files is the only way to do it.

And if I want to release a MacPython-OSX that sits on top of 
Apple-supplied /usr/bin/python in MacOSX 10.2 then I really must have a 
way to prepend to sys.path, because there is at least one Mac module 
(macfs) that is braindead in Python 2.2.0.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From greg@cosc.canterbury.ac.nz  Tue Dec 17 21:42:04 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Dec 2002 10:42:04 +1300 (NZDT)
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEMIDCAB.tim.one@comcast.net>
Message-ID: <200212172142.gBHLg4K17376@oma.cosc.canterbury.ac.nz>

Tim Peters <tim.one@comcast.net>:

> under FixedPoint's unusual "unbounded
> before the decimal point, fixed # of digits after, and when mixing precision
> the result gets the larger of the input precisions" rules, all of the
> following are always exact operations:
> 
>     + - *

This is true for + and -, but not * -- you need the
sum of the input precisions to avoid losing information
in that case.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From Jack.Jansen@oratrix.com  Tue Dec 17 21:43:23 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Tue, 17 Dec 2002 22:43:23 +0100
Subject: [Python-Dev] Strange crash in import
Message-ID: <911129A1-1208-11D7-BC1B-000A27B19B96@oratrix.com>

I'm getting a strange crash in import, and it turns out that there's a  
refcount 0 object on sys.path. Has any of the code for zip imports been  
checked in yet? Otherwise, does anyone have a clue what could be going  
on (i.e. what silly mistake did I make:-)?

The situation I'm in (the stack trace is below) is that I'm in the  
tp_init routine of an object when a macos-error occurs. I then call  
PyMac_Error to format the error. PyMac_Error then needs to import a  
pure Python module (which will give it the pretty huge list of MacOS  
error codes and the corresponding strings). This crashes, because the  
second entry on sys.path is a refcount 0 object:

#1  0x003fcf10 in find_module (realname=0x46f2cc "", path=0x13b310,  
buf=0xbfffeac0 "macresource.pyc", buflen=1025, p_fp=0xbfffeed0) at  
../Python/import.c:987
987                     if (PyUnicode_Check(v)) {
(gdb) p _PyObject_Dump(path)
object  : ['', <refcnt 0 at 0x13e548>,  
'/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/plat- 
darwin',  
'/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/lib- 
tk',  
'/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/lib- 
dynload',  
'/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site- 
packages', '/Library/Frameworks/Python.framework/Versions/2.3/Mac/Lib',  
'/Library/Frameworks/Python.framework/Versions/2.3/Mac/Lib/lib- 
scriptpackages']

If I manually import the "macresource" module beforehand everything  
works as expected, and sys.path[1] isn't a refcount zero object.

Here's the full stack trace:
(gdb) where
#0  PyType_IsSubtype (a=0x0, b=0x457d44) at ../Objects/typeobject.c:704
#1  0x003fcf10 in find_module (realname=0x46f2cc "", path=0x13b310,  
buf=0xbfffeac0 "macresource.pyc", buflen=1025, p_fp=0xbfffeed0) at  
../Python/import.c:987
#2  0x003fe520 in import_submodule (mod=0x4522b8, subname=0xbfffefc0  
"macresource", fullname=0xbfffefc0 "macresource") at  
../Python/import.c:1998
#3  0x003fe034 in load_next (mod=0x4522b8, altmod=0x4522b8,  
p_name=0xbfffefc0, buf=0xbfffefc0 "macresource", p_buflen=0xbffff3d0)  
at ../Python/import.c:1863
#4  0x003ffef0 in import_module_ex (name=0x0, globals=0x457d44,  
locals=0x1b6, fromlist=0x13b390) at ../Python/import.c:1714
#5  0x003ff0d4 in PyImport_ImportModuleEx (name=0x145b9c "macresource",  
globals=0x148780, locals=0x148780, fromlist=0x13b390) at  
../Python/import.c:1755
#6  0x003ce2fc in builtin___import__ (self=0x0, args=0x457d44) at  
../Python/bltinmodule.c:40
#7  0x00370b98 in PyObject_CallFunction (callable=0x12efd0, format=0x0)  
at ../Objects/abstract.c:1675
#8  0x003feb40 in PyImport_Import (module_name=0x145b88) at  
../Python/import.c:2160
#9  0x003fefb8 in PyImport_ImportModule (name=0x0) at  
../Python/import.c:1686
#10 0x004108a0 in PyMac_StrError (err=-43) at  
../Python/mactoolboxglue.c:107
#11 0x00410e00 in PyMac_Error (err=-43) at  
../Python/mactoolboxglue.c:169
#12 0x00265af4 in myPyMac_GetFSRef (v=0x18c760, fsr=0x13e4f8) at  
/Users/jack/src/python/Mac/Modules/file/_Filemodule.c:2400
#13 0x00263ef8 in FSRef_tp_init (self=0x13e4f0, args=0x457d44,  
kwds=0x1b6) at  
/Users/jack/src/python/Mac/Modules/file/_Filemodule.c:1243
#14 0x003b407c in type_call (type=0x268454, args=0x14fed0, kwds=0x0) at  
../Objects/typeobject.c:438
#15 0x00374028 in PyObject_Call (func=0x0, arg=0x457d44, kw=0x1b6) at  
../Objects/abstract.c:1675
#16 0x003da754 in do_call (func=0x45e88c, pp_stack=0xbffff8fc, na=1,  
nk=4443236) at ../Python/ceval.c:3398
#17 0x003da53c in call_function (pp_stack=0xbffff8fc, oparg=4554052) at  
../Python/ceval.c:3268
#18 0x003d8044 in eval_frame (f=0xa37c0) at ../Python/ceval.c:2009
#19 0x003d9354 in PyEval_EvalCodeEx (co=0x141660, globals=0x457d44,  
locals=0x3fcf10, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0,  
defcount=0, closure=0x0) at ../Python/ceval.c:2554
#20 0x003dbfe0 in PyEval_EvalCode (co=0x0, globals=0x457d44,  
locals=0x1b6) at ../Python/ceval.c:478
#21 0x0040b788 in run_node (n=0x0, filename=0x457d44 "",  
globals=0x148780, locals=0x148780, flags=0x0) at  
../Python/pythonrun.c:1089
#22 0x0040ac88 in PyRun_InteractiveOneFlags (fp=0xa390c,  
filename=0x141660 "", flags=0xbffffc50) at ../Python/pythonrun.c:593
#23 0x0040aa80 in PyRun_InteractiveLoopFlags (fp=0x44aaac,  
filename=0x137140 "", flags=0x14a4c0) at ../Python/pythonrun.c:522
#24 0x0040c604 in PyRun_AnyFileExFlags (fp=0xa0000cd4,  
filename=0x444fdc "<stdin>", closeit=669964, flags=0xbffffc50) at  
../Python/pythonrun.c:485
#25 0x00416db0 in Py_Main (argc=-1610609452, argv=0x0) at  
../Modules/main.c:442
#26 0x00001b5c in _start (argc=673326145, argv=0x0, envp=0x44653c) at  
/SourceCache/Csu/Csu-45/crt.c:267
#27 0x000019dc in start ()

--
- Jack Jansen        <Jack.Jansen@oratrix.com>         
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma  
Goldman -



From greg@cosc.canterbury.ac.nz  Tue Dec 17 21:50:20 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Dec 2002 10:50:20 +1300 (NZDT)
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEOADCAB.tim.one@comcast.net>
Message-ID: <200212172150.gBHLoKb17403@oma.cosc.canterbury.ac.nz>

Tim Peters <tim.one@comcast.net>:

> [Greg Ewing]
> > Worse, it would completely lose the connotation that it is a *decimal*
> > number type, which surely is the main point of it in the first place!
> 
> Only to a compsci type.  The intended audience never heard of binary
> fixed-point -- fixed-point means decimal to them as surely as New Zealand
> means anarchy to an American <wink>.

But they're going to have to learn that there's something
other than decimal in order to understand why they should
be using this fixed thingy in the first place, instead
of using floats. I don't think you'll be doing them any
favours by using terminology that obscures the difference.

There's a precedent for this in SQL, which has a type
called DECIMAL. Now, why do you think they called it that
and not something blander like NUMBER?

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From Wiktor Sadowski" <art@wiktorsadowski.com  Mon Dec 16 22:13:34 2002
From: Wiktor Sadowski" <art@wiktorsadowski.com (Wiktor Sadowski)
Date: Mon, 16 Dec 2002 23:13:34 +0100
Subject: [Python-Dev] Zipping Zope3
Message-ID: <004b01c2a550$60608460$0201010a@wiktor>

This is a multi-part message in MIME format.

------=_NextPart_000_0047_01C2A558.C1C03720
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_0048_01C2A558.C1C03720"


------=_NextPart_001_0048_01C2A558.C1C03720
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

[Just van Rossum]
>Ugh, you should really consider getting a decent diff tool :-/ How can
>*I* test if if I have no clue what exactly you've done?

Well, as you already know my implementation of custom imports (import =
hooks) is a bit different
and I am using  modified Python22 ,so it's not easy to make diffs.

Zipimport.c in attached zipimport.tar.gz  is to provide you with more =
details.
Don't try to compile it ;-)

>I can see that keeping the file object open could speed things up, but
>is it a polite & decent thing to do?

Yes, but it would be nice to not fopen/fclose for each =
__importer__.get_data.

Best
Wiktor
http://www.wiktorsadowski.com

------=_NextPart_001_0048_01C2A558.C1C03720
Content-Type: text/html;
	charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1250">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV>[Just van Rossum]<BR>&gt;Ugh, you should really consider getting a =
decent=20
diff tool :-/ How can<BR>&gt;*I* test if if I have no clue what exactly =
you've=20
done?</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Well, as you already know my implementation of custom imports =
(import=20
hooks) is&nbsp;a bit&nbsp;different<BR>and I am using&nbsp; modified =
Python22=20
,so it's not easy to make diffs.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Zipimport.c in attached&nbsp;zipimport.tar.gz&nbsp; is to provide =
you with=20
more details.<BR>Don't try to compile it ;-)</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>&gt;I can see that keeping the file object open could speed things =
up,=20
but<BR>&gt;is it a polite &amp; decent thing to do?</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Yes, but it would be nice to not fopen/fclose for each=20
__importer__.get_data.</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>Best<BR>Wiktor<BR><A=20
href=3D"http://www.wiktorsadowski.com">http://www.wiktorsadowski.com</A><=
/DIV></BODY></HTML>

------=_NextPart_001_0048_01C2A558.C1C03720--

------=_NextPart_000_0047_01C2A558.C1C03720
Content-Type: application/x-gzip;
	name="zipimport.tar.gz"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="zipimport.tar.gz"

H4sICNpL/j0AAHppcGltcG9ydC50YXIA7Tz9UxvHkvlVqcr/MJbLIIEAQcDOmYdTshE2d9hQQPz8
YlNbizSCPa92ld2VbWLzv19/zOfuSohccq/qKsp7lnamu6enp7unp6eX36NJNJ6kWbE++O6v+ggh
njx5IuAbPt3St9h+0t3ZAYidJ0+2t3ce72xD2+MfdwC++93fn7/68zBKBvF0KEXz5Ka4TpP16+YP
39vGvMimg2Isx5cyK3Wl+VCO8lLjOMzy6zAutQ7S8SSKJbXif3kRFtFADK7DbEVMbgL5pZBJHqVJ
/v5C7ImvzfXJzaDZEfidqu/m7S4jb6wIo7QyE+nlf8tBIYCXKIkKoCHCZCjy6QT7xcoGohQ3EwkA
gmfjof8aTQ7Vb6JfA/L1h+8bJzfHNE7wqt/bt88wQJgNrqNPcleIDZhLWFwn4ViKdCSKa4lkhAIA
Vjy8EQgkByzCG0bQdCUTmYWxwB4RJaO0jDLJYJJfdgmFobjlqWiGG/n0cmMYZQCYZjcbTcJF6h6n
iATTvN21qwAA5yAeBnLFEWAzAeJoafY5zIYg5kEMpA2ypX4li6BIBy1cVCOV9u4MSBBMMAyLsKWb
V0LkreNhd2D2OAUQRiFAh2C2ed6hJ8QN8uh3yY+IW35MR6NcFshBDQ9xGg6JiXE6nMayZecNw+cy
HnUEszKaxjGsKPHCLYCBy8wjRfkkHHwMr6R+VKo+7DiDgVgCmRTZDTPzkHRVerJ+cS0HH1vppG3Q
SPy6uSOWyivTrp3XD987mhsk8nPLW96VQZx3PJW48p4/fh7mQBg1Hhb98CpJMykAaDoG/gHSJY7m
Jorwo8zFIMyUyocFLxoZ4WcphmmyXMADQKZiDMAijkZSgCCHMkOtEqC3gzjMc5mvs75XloKE1sBf
4Bv8lWrDfNaeFZMgjGPQPZpdF6XciEaiRSgP9sSbX46O2mTGRGXtmTZJ7tq1HWSU1WY2M6f9Fv6f
yWKaJaJlpddW3N7apUGt+OH7stiq6rbgorAGovrRHEGssUzUo5r0g5ObXnYVnIAjlufTCeg202se
P3jq8LEeECdB0OzgPBuCPksnN2dFFiVXpGKgdZ682m0A1fNe2+QxT26CwzcvTvsHLR8WGUTOQGqG
aC/nHzWgMI8y9BnYcwnSTlPDP9sTr3vvTnrnr476b3iRgaV+lgVnslCjwfOXQcACh5404yk3mloP
iFKRpiJOk6smjFKaZoNWtKQj2uEhslE5D4T1RSwtiQfMErAhJ7hHvQ6LwbXMNWfHxFVJvHNI8jzR
RPUMxuGNuNTbDtrVGAEEf5obyONGkW4o8HVQBNwwYL9ostE5uwzCBlpNQHFW9G9H/1kRL6ej91b4
q5sXpguoAGLFbBoTeNDa20Drb+3u8mRoNuAWhzQBcAiFSAeDaZaFyYDm1USmFbdECHdz+F8LRuK+
Nu/B7969E2Gcp2L918MTBU+CnBjx8fJfZjL8uGu7329fIMRZ/6RtCH37JnpH59CkCTXMpCa7ipPV
PbGNv1FLmBTD6OGAiGlao+m3ocuoMLRttx0hJKnA6cAiTkEaacZPUS7CQkhoUYxYPUFelQohWF5E
cSxg61OQt9YkXsQyzFqkrNbgSryJVbHNENyM09uhZxB2MpjctGDZOwTaQe+jcDYZB1VCNWIkt/yh
u1waDajAgxqTkazI3luZbF6IB7QYWjQMYalT567XZbnRg9t1UcMDg7uVjcD1OgdZOuafbNmVbWAG
sJ2QnlJpCKsMPjWrk77xKws/z25w77yC9QwxJgG/MfTC0xHwIPKbfN033HXlAP5pNuHhEAiReQ2Q
ikinGXDyyWy8DZ8Az/MmD17Kgn1Dq+lDNO3ylVAfGNcHNPYhuOUApwRmtN56GAWOYx4WclxCUH4b
51XdQZgRDeuwUA20DJTmgCSdJ+iVKRbPSapWVkN1zFCCKocLrVJUYsgrEGYPt8p3eq+0+Ir529m2
esc2cNe+NBsfOj2lIxb88KZbimc+pdHQD2iGkoKvakzDMQvNer8/I0KwnS6D9V2Oy+CG9DLAkx0F
gKNMcrDQrgvAMPIMkDSE8nOCfezVvy+npnllEtiIHx5M0D8vKmMLVtuI3SWBJCggOAx1NEWvqp3q
nYGSjpSMG0U0bNR+13pFaFNsQ5P1fWIVeYN/dXjhybYULFvWLNoMJu3ysMoRdCmW8+FUWDVjEFyJ
ymytSuMPb762adX4fj3Dk5vX4WSCXLwK8/+SiiVX33g7087AXWEgRZLSbCKcy2fThNHevP9Dz07G
uayl2iVwOw/LsjU+pUa3C5z0MHLSZ1kLA06rfLBQKlsxgerRCpB3jXJDsMjGsUKWUAr9LrTeRxQS
dOyxeIFzSe4dS5yJ+CcT5sA7g5Cu2hFgicBOspbhdXl9ua1Veu7R4G0YT6V3MlA0xHiaFxCRFRhc
h7CRFoUcrmOPd0pwz4VGZ32fw27GsEbmvsSyXDLSMscIpPAP0a2dK8fNEemQiADsR/haXeUJVvW0
U0qwRRc0yh+yjNJhjwnNOgjb3YT9IC8qCpNDW95LLUX49SZNSAaapGqiaS9mB+lEJosYAKrUfW1g
hEuHj/Rj72SBU61pAWEixgF8uUEjUeo0s0tWJysMF9ETidfBzmEBsUBMQOm2RURzb9+gU2TznEDD
T+HZ9J2TunPTdohiUoME7rbYjJrmQp+EF1qW+3kkLTvfHeGkZ/sjBDE8gviMoenAts7SDMndmQwa
msBlBJ8meA8r1SUjVp9VK+ElR8Rs1FUXxnqoIi+butTzwjFgRm4S19fWDkva8nCfdXeWB2F1pFor
ZN2AgAtbAiR+/9KN0uSLzZ5Z3Sy9kPHP2zqdqd1HV3kIbTZ8QCrZDBxaOatbDhftGatEoBQT3nP3
VeNSElkNFRaAcjktZI57Mu3HORHJ5+3EaglgzXRwgLIw0YGxNANn1c3BNQ5OBXRO7+oqJ53yj9EE
KdKm9icEARsrJHAduetUjMYV71chq2dizwvogS2eUaoRM7XjaNCuRbuKcTP8iwB4Vti4K7Couc5Q
mb0O3YnYNN89whAEvbefRKI2hrHoD/xTsX8fg2YbIciWilqY5dI9EA2kPLFdIrM0pJ/arrzLHsMF
U6/1oPYsTmPv6fDGzUFchkPQmasoEckU71wF/Lq8KST40CGwg9NM5BeTh7DjIE01QmOQJkWUTKVN
LBjzAKjdmYkGbF0geS6axus8hfRIQkJU4R0ZY8kWgWrJNb+WcNU83Jcj70JpTK3qChiofG16hwEP
1vaAVF73z18Fb3unvdOXZ8AdHLibtx0iAJzNIAA9ixDAeLKEiU3zUGy8UBeBzcEkW2o08OuW8lES
l1HifWvlyvY13chXJEitrgTZ3HHy4jw4fv6f/RfnHcFbbTpy97GO0M6+g8UZai5kcYh8NzqBErJF
Z7fV7NyJrPaJjkElIehZQ04QjOjs/BSk1vJST+kAoFGEAn7SdWK1NKDp37ru9w/6p6f9/aC3v3/a
Pztr4XdbdBe9ERd71YIAiNoPz1sVyktMCNF47+3i3Kz1rDvM0o6N4ZAvmnaH0BrKPRSQRACviHDs
AlpDyRULcH1Uk5TrMJJ+BJwyuQkYeVFtR33FLbemJ5/Zg+4Qb4KrPZmc1CGEeUBurr4PRvptKuH6
p753zDtFTed1mF/XsQcyqJuPmotd0pdYiBENYAvqwUQ7JZGks2WS1nMKm8hImkGC85ODo97LswD0
pffL0bn4JpzG572z/vm/TvodweijOLzKGbWs+WaIYe3KFln4CZyBrJMEuvyadpj09exF/AzXZXGU
F2zCNQARcDaj2W5aNR7fzEQ3MGTVs1lI3VCvufXsXYa1wsB8f12zzAfotmf3qVGq5DwJtTBGm2Tp
oF2uAtCKhQ8KmB2GVsAem7AGc0y4VORhYPCBIQKjzfsy7ij2VJ6cIJxyKiAVS6zwCPFuGnq5HY97
5jbBHAXWnrGrV8fBl9AIJVf6Ch1+2Zonip+ieAiNKJWOUCZLBCjFmxONFswrjNXllMWd5gBLYWqY
A4lcgl6G6OgIEkSBYVUE11eKlZ6Nwui6FNNM2VqBB6anBMKflj2O0uehOZ/i5D9CYLErurSXTBPd
I4cG3TnDKnT4TTfUFpgPyUBtGOUfDSZNuxZzKEu4JRzWpw7h8G/4IkB1+1eEGTZp2Rn0tpr3PheG
obQwYsPTEzSkVwkwMIQAkwQPDiOKtcijhNpU6ZMzbTqVuWwJ9BZdWISVjRnn79rSMHW+tnAqul+h
3M/B4VEfmib4G4synBzCIBsskEggEIyYo7E0RK5lCGVHGgInpzBVDz4YaDqR06H8Cg9oA8hZFvro
TurunOnhjLNjzvVEd2bnysTLWAN5a1vPFq1lOQbPPorTz96pGQyP+KLxwQJMRcvME7J7TLMBoMoy
T/DkS/lUY5GcqFTMI0BNbv0Aak/CwlS21B4Z8DoaKaOvoEWDCkabsGguP1rf6nbz5abhaeYEYCHl
R2ClI9a2tjp4uEM/dwaNbBkSSyhGOAZ6PXshxGWqwSks+xFICFOymJsFSnRj0P3Sfdzd2X6+07Wl
Ns/D4VPRZ3ov0MuEMdoVGPFVEgJjaoDGaBCnOVHavY9UMC8e/q8FAv+SHdmb9DdQA2jWjPrmJNV8
iT7uqNOyZzdAeq4AEYMWQTklKyxdncqCYsPCcznqaLt8xUf2pi7K/FIhy2LZoLtcAmSHxvkqILU6
8ULs8yWM0oSt7iZpArTroiGtD54iHOBgr/RgcxjFq8MucYtA2rNVGDu7BsWocIYObUFQ8ICLgoJT
XUwwxvEuBo4LcA9w7ZEXZZvFqjG2H4NkLYlVroC5i9CicLvzV3R7C1bUTvo+xkLaZvj2dwHo9sVi
+wBV19Tp7GQlhWcw3VReYwVxaDtuw9480HwwI9gJt9Ebyzi2BralVqurOoGkurpkDT2sLpLZOAJf
iDFEAik0ylwpUyiJa6+04yofS7ljUyqcg1OlKITiQ8+Yoxw25Az9SdnUCMz1CH/Q6re7P2qrtxk1
a/hHyFLJ5Kt8WPXYeuwx82MX2v4sxQSmWERaOio/YHTRSj0G4LJ/RJMmaLci4cqU2NnaP5vQZQBb
qtFxK1zWVDf82DRJXlqA4DkeByjf31L3V3ddDrEOenGwk/t1qur0fnfmJo55w9M8Fm1zqd64SkHL
JG7DQM3L1yqWeQNaXVUbq7utmzAieCuzS2g9gEM6keWCuX9mcOQ9K4ZAvtV8KOwmTzV4j+Ih8aNq
zB7lHxJK/qgRvS1eb8+6iowYfurd0FFXXdZ1TjBuzxvBCA45LSzqUvF4BcPAunXxtrZKXEOWIfg9
ji6he23TxrZOO3bgHZElVYoc7WiIQAuCP0hnOBMW8NdrztI3sbfJis+iFdjCKk+j42O5TM5wpKOK
mgw4LTVd8tQgbTKSJxFzzFbZIqV3iAHxnQVtli8GCMSOOUOjZqrU2OgU0rFa5LH8M7wY9AkOdeEl
5L8hpPzlTe9t7/Co9/yoz/zcmhstrzrRst32blpt+xwF+799vad8iszCzyrPvoJf9orK471havzK
h02kD0Lu0N6VBxk4VFSXzjANyAHsGRVv0ZTaX2kWo4mu7ujl+KU6d6mTLljRoesJzzpbKXhcEQ1q
xu266/X1rgMaLQ58q6HvPqHxVlM+p9GpBIcgmYOWLXQC4ZtGddxv02VF8JqLRM3C4t5hPCkhLFCn
BYTg+OAeeqvnShoHA1/OupI4vGPZHMZh4YHxWaGEYZfA9tRO4unJCL+5lnPT29BoWLskelXb5Y0F
/rNjfPvmauGDPUtw/osux/MlhNSIlPJJvEAHkBhEzuvFY/jSRoTs4OFYF/0b1qiU8ddldS0efoYA
DoK1iM76KE94lVP5aYNiN1ltvfU19r1kiHWkLF+DjQxX5qC4RXp315G48+KFJeYJMcrJBmx+Dpm3
RPQAptp3324On6Pi2u5L/rZRtwsbrm3HQlbBW+P8NXdIEruiqSCRP5qh3SiQDzdA0ouDiO6O9wJM
7QAYx6Sx5ZleNYvBT1hPvLa5gzR57ytvMPVRjUaG7hl1PjN2n2kyVrEy3pi3ykU5tu6Gny0qUq/s
JkjD2y5q32YjVL15oEqW32JjCJNYhJebA7zkH9gAh+KH19j2hi7EWk4hFFH8BxQV31cLoJgAx2Jb
v8OuW2jE3QuxBGef0YiSYC3LqG62hUAEv+nDtyz8s2dC/NSegbY1D23z8Sy0H+ehbW0bNBDTPQOq
56rqYuBUXXDKrWlLPrTSkCBn1K2WClfJCXKCMBZ4/Qiki5SqOMIYb8XCgt8MVxaG6lY5qbIuWi+o
SuV/6rCqrYmfjONA/JmuzlZ6vQAwLtMiDCUwa33UOit7SbfZbsygogahimF0JYgSoHagIVWw6Ct5
jWqlOks9VRNhzwk6oXogiOF6P4CELxaFtWxnDrdAA/9Vjd+wjr3WlBWw8x7HVfWlVANVur/4IocB
d7Hxui3zd7SJvmfwwiIXf/4O9lvdZDx8nZWFm/g4BHLND9mH5OcmBBjwM8H3KktpVi8Z9CFbNrmj
31ZXBTYly07OqDUhzhk4UcA2XQQfnt/aGj/eemcthzDej5QH7y67uVf8OTEhg8sPmRuese1V1ge8
ziQo5w2LVkvLyhSse6KiN4uUH7fL4uxEDvQ8pVQWwYD32YsYoXY36iygaBVj4J4/oFfGIQUveC5K
s+7QNaeCzpSpUSwdJZNp4ccCJS2tt/gZAv43/RWH8rpQlKOWhyjDv3jhjZPxTrwLvgt//1p5tbCK
yN3R7r+rspwY+OsryBesH0cou9bMtFb5UhxprdYoDtL3kigevmf7d+NbczDNdkO3OU79rGMOOy3q
stMyGq1mpfxiENALtYGqtdBb9cqlBNcvV7CNx8AiGrVHyy9yMC30wUcnSJxX/fFlfSjOMC1mlFLR
O/bDgG7QC7cGKqXnvvxgimT9409thFLRMhXTqjeOVP10ym9Hr8MfPKBCbNXT5jnpa3DNt1kkHkHN
zxaRW0DMj5f+7ID6ewE/O8lITeCB+9rtim5e3WzbvQm70HWQkFg0eDLAzLap8OU0RX2+m+tsmppB
J3wtp9/f5xdg4xqwbeUO1IHT2jy5km31LbDyuvZBZzDK5Bn0v2xZAyD9smagKeB/pfW9Z6JdJYIf
5eIhVSSDxj5SOXZwM26G1PJirR9zpZhswYGbAnKlbLhNw6mXCcXG+oMrdNyZjHfKn4Fpv+65Ughc
rQSu1sQimbnFsKY6m3D3j4L+u5Pj03Od+McCNANLjaU9rrSrmXXBswAdUm5a1T9QVHqD0cULjidw
jx39TgtqXdSpVBWL+Pe/sM6IfmFVpfU2xZg9DXyLvdJLBd0L6io3VuA26+DovWFFHdfLKnVwCPJh
Vd5u2dxKs+OtHzzZZSCTU6nnk3+dvzp+E/RODoO3/dOzw+M3uAx3mjlKimDsHqbMcQTBrZXmPD/g
FRh7b2jWL1dlDPxCBWcd/u7vz9+fvz//nz7/A0HRP5IAVAAA

------=_NextPart_000_0047_01C2A558.C1C03720--



From martin@v.loewis.de  Tue Dec 17 22:11:36 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 17 Dec 2002 23:11:36 +0100
Subject: [Python-Dev] Re: [Zope3-dev] Zip import and sys.path manipulation (was Re: directory hierarchy proposal)
In-Reply-To: <200212161747.gBGHlYK03859@odiug.zope.com>
References: <Your message of "Mon, 16 Dec 2002 10:15:01 EST."
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <20021210103937.GA21447@vet.uu.nl>
 <15862.10111.488361.591077@gargle.gargle.HOWL>
 <20021210191501.GC23933@vet.uu.nl>
 <15862.16384.979878.277669@gargle.gargle.HOWL>
 <20021211083643.GB25606@vet.uu.nl>
 <15863.10433.412101.910639@gargle.gargle.HOWL>
 <3DFB261B.2000709@zope.com> <20021215072350.GA12339@vet.uu.nl>
 <3DFC960E.1030204@zope.com> <20021215211435.V11257@prim.han.de>
 <5.1.1.6.0.20021216100402.02f217d0@telecommunity.com>
 <5.1.1.6.0.20021216111815.02f22c90@telecommunity.com>
 <200212161654.gBGGsRe03537@odiug.zope.com>
 <15870.2216.65988.322683@gargle.gargle.HOWL>
 <200212161747.gBGHlYK03859@odiug.zope.com>
Message-ID: <m31y4ggusn.fsf@mira.informatik.hu-berlin.de>

Guido van Rossum <guido@python.org> writes:

> - Nobody remembers why there is a special case in site.py for lines in
>   the .pth file that start with 'import', except that Martin von
>   Loewis once wanted this.

It serves the deployment of packages: you can have code that is run
from site.py, by just installing a file. This is much simpler than,
say, modifying sitecustomize.

I specifically wanted this to register new codecs, and, to some
degree, it is used for that. The Korean codecs package ships a PTH
file that says

import korean.aliases

korean/aliases.py starts with

import encodings.aliases

encodings.aliases.aliases.update({
    'cp949':        'korean.cp949',
    'ms949':        'korean.cp949',
    'uhc':          'korean.cp949',
    'euc_kr':       'korean.euc_kr',
...

Adding new codecs or new aliases is very similar to adding new items
to sys.path. I guess you can also use it to register new importers,
if Just's future PEP is accepted.

Regards,
Martin


From just@letterror.com  Tue Dec 17 22:13:47 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 17 Dec 2002 23:13:47 +0100
Subject: [Python-Dev] Strange crash in import
In-Reply-To: <911129A1-1208-11D7-BC1B-000A27B19B96@oratrix.com>
Message-ID: <r01050400-1022-DB02EC2C120C11D7A7C4003065D5E7E4@[10.0.0.1]>

Jack Jansen wrote:

> Has any of the code for zip imports been checked in yet?

Nope, nothing. I would've loved to take the blame ;-)

Just


From greg@cosc.canterbury.ac.nz  Tue Dec 17 22:14:22 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Dec 2002 11:14:22 +1300 (NZDT)
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212170639.33487.mclay@nist.gov>
Message-ID: <200212172214.gBHMEMZ17581@oma.cosc.canterbury.ac.nz>

Michael McLay <mmclay@comcast.net>:

> Let's go easy on the kiwis. They don't always get their due. It
> looks like "Mad" Pearse [1] may have flown before the Wright
> bothers, but since he was a perfectionist he did not try to claim
> the title. <wink>

Most people seem to agree that he managed to get off the ground, which
was a non-trivial achievement at the time, but it wasn't what you
would call a controlled flight.  The Wrights were still the first to
actually make a plane go where *they* wanted it to go. :-)

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From just@letterror.com  Tue Dec 17 22:18:20 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 17 Dec 2002 23:18:20 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <004b01c2a550$60608460$0201010a@wiktor>
Message-ID: <r01050400-1022-8302526A120D11D7A7C4003065D5E7E4@[10.0.0.1]>

Wiktor Sadowski wrote:

> >I can see that keeping the file object open could speed things up,
> >but is it a polite & decent thing to do?
> 
> Yes, but it would be nice to not fopen/fclose for each
> __importer__.get_data.

It also does this for each import (I copied this behavior from the PEP
273 implementation). I experimented with keeping the open file object
around, and performance hardly went up.

Just

PS: your computer thinks it's monday.


From tim.one@comcast.net  Tue Dec 17 22:40:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Dec 2002 17:40:17 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212172142.gBHLg4K17376@oma.cosc.canterbury.ac.nz>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEFMEEAA.tim.one@comcast.net>

[Greg Ewing]
> This is true for + and -, but not * -- you need the
> sum of the input precisions to avoid losing information
> in that case.

Yup.  The module docs have the full story.


From tim.one@comcast.net  Tue Dec 17 23:03:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Dec 2002 18:03:56 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212172150.gBHLoKb17403@oma.cosc.canterbury.ac.nz>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEFOEEAA.tim.one@comcast.net>

[Greg Ewing]
> But they're going to have to learn that there's something
> other than decimal in order to understand why they should
> be using this fixed thingy in the first place, instead
> of using floats. I don't think you'll be doing them any
> favours by using terminology that obscures the difference.

We were talking about the name of a hypothetical builtin function, not about
documentation terminology.  If you want to presume that users are idiots as
well as simply ignorant, then fixed_point_decimal() is what you want.
"decimal" leaves out the vital distinction between fixed and floating
semantics, which is more important to "good to the last penny" results than
the distinction between binary and decimal.  A short name simply can't
capture all that's relevant (and I don't care).

> There's a precedent for this in SQL, which has a type
> called DECIMAL. Now, why do you think they called it that
> and not something blander like NUMBER?

You can't take any useful lessons from SQL here.  Most variants of SQL have
both DECIMAL and NUMERIC types, usually aliases, sometimes subtly different.



From just@letterror.com  Tue Dec 17 23:09:35 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 18 Dec 2002 00:09:35 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <r01050400-1022-8302526A120D11D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <r01050400-1022-A322CE2E121411D7A7C4003065D5E7E4@[10.0.0.1]>

FWIW, http://www.python.org/sf/652586 has been enhanced with the
mod.__importer__ feature.

Just


From greg@cosc.canterbury.ac.nz  Tue Dec 17 23:41:54 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Dec 2002 12:41:54 +1300 (NZDT)
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHEEFOEEAA.tim.one@comcast.net>
Message-ID: <200212172341.gBHNfsr18263@oma.cosc.canterbury.ac.nz>

Tim Peters <tim.one@comcast.net>:

> If you want to presume that users are idiots as well as simply
> ignorant, then fixed_point_decimal() is what you want.
> "decimal" leaves out the vital distinction between fixed and floating
> semantics, which is more important to "good to the last penny" results than
> the distinction between binary and decimal.

I don't agree with that. The first and foremost reason why a
penny-adder will get the right result with this type whereas he
wouldn't with a Python float is that it's decimal and not
binary. Floatness only becomes an issue when the number of dollars
he's adding up becomes large enough. If it doesn't, he could do the
same calculation on his floating-point-decimal calculator and get the
right answer, too.

So, no matter what it's called -- fixed, decimal, or even
fixed_decimal, he's not going to be able to guess that he needs to use
it, he's going to have to be told.

Given that, there's no point in worrying about what naive users may or
may not think when deciding on a name for this. And I think a name
should be chosen which reflects what it is about this type that makes
it useful. I agree that the non-floatness is important, but it seems
to me that the decimalness is at least as important, too.

You're going to have to face this issue if you ever want to add a
floating decimal type as well. If the fixed decimal type is just
called "fixed", what are you going to call a floating decimal? You
can't just call it "float", because we've already got one of those!

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From pobrien@orbtech.com  Wed Dec 18 00:23:05 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Tue, 17 Dec 2002 18:23:05 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <m3znr4haow.fsf@mira.informatik.hu-berlin.de>
References: <200212161609.03192.pobrien@orbtech.com> <200212170912.11638.pobrien@orbtech.com> <m3znr4haow.fsf@mira.informatik.hu-berlin.de>
Message-ID: <200212171823.05716.pobrien@orbtech.com>

On Tuesday 17 December 2002 10:28 am, Martin v. Löwis wrote:
> Your original message sounded much like a support question. I was
> trying to say (perhaps in impolite words, for which I apologize) that
> posting support requests is not appropriate for this mailing list.

Apology accepted. No hard feelings. Sorry I rubbed you the wrong way as 
well. That wasn't my intention.

> And it is a good thing that you have submitted a bug report to
> sourceforge.

Thank you for simplifying my example even more, and describing why 
cPickle is doing what it's doing. Of course, we're still left with the 
question of whether or not this should be considered a bug. I think it 
is, but I couldn't tell from your description on SF whether you feel 
the same way or not. Certainly the difference between pickle and 
cPickle is disturbing, is it not?

> It was the (perceived) attitude "my bug report is more important than
> the other 320 open bug reports" that rubbed me the wrong way. I do
> appreciate that you took the time to report the bug.

I honestly didn't have that attitude. Sorry if my message sounded that 
way. I just wanted to try to resolve this while it was fresh, that's 
all. And I do consider it pretty serious that pickle and cPickle are 
not working identically (in substance, not cosmetic differences). But I 
don't mean to imply that this bug is more or less important than any 
other. It just happens to be important to me. ;-)

Thank you for your help. It is greatly appreciated.

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------



From tim_one@email.msn.com  Wed Dec 18 04:44:43 2002
From: tim_one@email.msn.com (Tim Peters)
Date: Tue, 17 Dec 2002 23:44:43 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212172341.gBHNfsr18263@oma.cosc.canterbury.ac.nz>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEPJDCAB.tim_one@email.msn.com>

[Greg Ewing]
> ...
> You're going to have to face this issue if you ever want to add a
> floating decimal type as well. If the fixed decimal type is just
> called "fixed", what are you going to call a floating decimal? You
> can't just call it "float", because we've already got one of those!

So if the fixed decimal type is called "decimal", what are *you* going to
call the new floating decimal type?  Correct me if I'm missing the obvious,
but it seems "float" is just as forbidden to you then.  At least if the
fixed-point type is called "fixed", it leaves "decimal" available for its
successor.

But it's all hypothetical so long as FixedPoint/fixed/decimal remains just a
class in a library module.



From Paul.Moore@atosorigin.com  Wed Dec 18 09:37:55 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 18 Dec 2002 09:37:55 -0000
Subject: [Python-Dev] Zipping Zope3
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com>

> FWIW, <http://www.python.org/sf/652586> has been
> enhanced with the mod.__importer__ feature.=20

... and I've just added a comment expressing some concerns.

I think we need a PEP about now, pinning down the "abstract"
import hook interface. What zipimport does risks turning into
a de-facto requirement, before we have any other hooks to
compare with.

My specific issue is with get_data, which is totally unneeded
for the import protocol itself (although it does solve real,
related issues). I don't think it should be a requirement of
the import hook API. Documenting it as an optional interface
which hooks may choose to support may be reasonable, though.

[I can construct theoretical use cases where it could be hard
to support get_data. But I don't like theoretical use cases
any more than anyone else - I'm just suggesting that we wait
for practical experience before adding anything more to what
is currently a nice, clean, and simple API]

Paul.

PS If Just doesn't want to, I'll volunteer to write such a PEP.
   But be warned, if I do, I'll probably indulge my instincts
   and make the API as minimal as I can get away with (I *like*
   the iterator interface :-))


From mal@lemburg.com  Wed Dec 18 09:43:17 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Wed, 18 Dec 2002 10:43:17 +0100
Subject: [Python-Dev] Deprecate doctest?
References: <BA1FF2F9.2D123%goodger@python.org> <3DFB101D.6010306@lemburg.com>	<20021214194215.GA9536@panix.com> <200212161624.gBGGO2j01562@odiug.zope.com> <20021216173642.GA10468@panix.com>
Message-ID: <3E004335.7090500@lemburg.com>

Aahz wrote:
> On Mon, Dec 16, 2002, Guido van Rossum wrote:
> 
>>Aahz:
>>
>>>That's tricky, given that doctest is in the standard library, and that's
>>>exactly what doctest does.  Are you advocating deprecating doctest?
>>
>>You gotta be kidding.
> 
> 
> Of *course* I was, but I couldn't figure out what else MAL meant.

Inever talked about doctest; only the fact that error message
strings should not be relied on since these tend to change
rather often between Python releases. It is also possible that
some of these cat get localized (I remember having seen Martin work
on something like this in the past).

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From just@letterror.com  Wed Dec 18 09:58:52 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 18 Dec 2002 10:58:52 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-969E3258126F11D7A7C4003065D5E7E4@[10.0.0.1]>

Moore, Paul wrote:

> .... and I've just added a comment expressing some concerns.

Likewise ;-)

> I think we need a PEP about now, pinning down the "abstract"
> import hook interface. What zipimport does risks turning into
> a de-facto requirement, before we have any other hooks to
> compare with.

True.

> My specific issue is with get_data, which is totally unneeded
> for the import protocol itself (although it does solve real,
> related issues). I don't think it should be a requirement of
> the import hook API. Documenting it as an optional interface
> which hooks may choose to support may be reasonable, though.

By no means it will be a *required* part of the protocol, if it'll be
part of it at all. I just quickly added it as it's simply a method
that's needed for zipimporter objects.

I like to look at the big issue the other way around: if we had a
Virtual File System protocol it would be a snap to add importing
capabilities. What we then need is Zip file support for that VFS, and
the actual import hook could be completely independent from Zip file
support: it would work for *any* VFS implementation as it's generic.

The importer protocol should be in no way hard-wired to any
file-system-like properties (and indeed it isn't).

> [I can construct theoretical use cases where it could be hard
> to support get_data. But I don't like theoretical use cases
> any more than anyone else - I'm just suggesting that we wait
> for practical experience before adding anything more to what
> is currently a nice, clean, and simple API]

Don't worry: I couldn't agree with you more.

> PS If Just doesn't want to, I'll volunteer to write such a PEP.

I've started, but I'm sure I can use some help. I'll send you my draft
when I'm stuck ;-) Thanks for the offer; we could surely do this
together.

>    But be warned, if I do, I'll probably indulge my instincts
>    and make the API as minimal as I can get away with (I *like*
>    the iterator interface :-))

The iterator interface is a great example for me, too: it's a piece of
art ;-)

Just


From martin@v.loewis.de  Wed Dec 18 10:25:16 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 18 Dec 2002 11:25:16 +0100
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212171823.05716.pobrien@orbtech.com>
References: <200212161609.03192.pobrien@orbtech.com>
 <200212170912.11638.pobrien@orbtech.com>
 <m3znr4haow.fsf@mira.informatik.hu-berlin.de>
 <200212171823.05716.pobrien@orbtech.com>
Message-ID: <m3znr3lj3n.fsf@mira.informatik.hu-berlin.de>

[Jim, this is about a perceived cPickle bug where refcount 1 objects
 lose their identity when on load they appear in multiple dump calls]

"Patrick K. O'Brien" <pobrien@orbtech.com> writes:

> Thank you for simplifying my example even more, and describing why 
> cPickle is doing what it's doing. Of course, we're still left with the 
> question of whether or not this should be considered a bug. I think it 
> is, but I couldn't tell from your description on SF whether you feel 
> the same way or not. Certainly the difference between pickle and 
> cPickle is disturbing, is it not?

I was happy to discover that this is a *really* rare case, so I
refrained from classifying it as a bug or non-bug. For example, you
can't trigger it by using cPickle.dump[s].

It is, strictly speaking, a bug, since the documentation says that all
objects will be recorded in the memo. The question is whether this is
a documentation bug (i.e. the documentation is promising too much) or
an implementation bug in cPickle.

There clearly is a documentation bug: objects like integers are never
put in the memo, and lose identity on unmarshalling. Of course, it is
not in the spirit of pickle that their identity is preserved. So any
correction would have to include a clarification of the documentation.

I'm a bit concerned about performance implications of changing the
behaviour: you'll have to put every object into the memo, even though
in most cases, you'll never need to lookup any refcount 1 objects.

It is not clear to my by what principle cPickle decides to use put or
put2. If cPickle is changed, the distinction between put and put2 goes
away. A cPickle expert might be able to tell, but I think there are
none left (which is also a reason for my initial reaction - you are as
much a cPickle expert as anybody else here).

To eliminate the performance concerns, it might be feasible to add
another flag to pickler objects, indicating whether this was called
from pickle.dump[s] or pickle.Pickler, adding the performance cost
only to people who use pickler objects. For that to be reasonable, one
should be sure that
a) the majority of users uses pickle.dump[s], and thus sees the old
   behaviour, and
b) refcount 1 objects are frequent enough to worry about the 
   performance hit

I would appreciate if anybody could provide data on either aspect.

> I honestly didn't have that attitude. Sorry if my message sounded that 
> way. I just wanted to try to resolve this while it was fresh, that's 
> all. And I do consider it pretty serious that pickle and cPickle are 
> not working identically (in substance, not cosmetic differences). But I 
> don't mean to imply that this bug is more or less important than any 
> other. 

Whether it is serious or not can be only answered once it is
understood: As you can see, I still hesitate to classify it as
serious. The reason is that cPickle always behaved this way, and
nobody ever noticed. This tells me:
a) almost nobody uses multiple dump calls to the same pickler, and
b) of those who do, almost nobody worries about cross-dump
   identities, and
c) of those who do, almost nobody ever ran into a case where a
   refcount 1 object occurred in a bad position of two dumps, and
d) of those who did, almost nobody noticed, and
e) of those who did, only a single person was worried enough to
   report this to the cPickle maintainers.

I'd really like to know what Jim Fulton thinks about all this.

Regards,
Martin


From aahz@pythoncraft.com  Wed Dec 18 12:36:39 2002
From: aahz@pythoncraft.com (Aahz)
Date: Wed, 18 Dec 2002 07:36:39 -0500
Subject: [Python-Dev] Deprecate doctest?
In-Reply-To: <3E004335.7090500@lemburg.com>
References: <BA1FF2F9.2D123%goodger@python.org> <3DFB101D.6010306@lemburg.com> <20021214194215.GA9536@panix.com> <200212161624.gBGGO2j01562@odiug.zope.com> <20021216173642.GA10468@panix.com> <3E004335.7090500@lemburg.com>
Message-ID: <20021218123639.GA28416@panix.com>

On Wed, Dec 18, 2002, M.-A. Lemburg wrote:
> Aahz wrote:
>>On Mon, Dec 16, 2002, Guido van Rossum wrote:
>>>Aahz:
>>>>
>>>>That's tricky, given that doctest is in the standard library, and that's
>>>>exactly what doctest does.  Are you advocating deprecating doctest?
>>>
>>>You gotta be kidding.
>>
>>Of *course* I was, but I couldn't figure out what else MAL meant.
> 
> Inever talked about doctest; only the fact that error message
> strings should not be relied on since these tend to change
> rather often between Python releases. It is also possible that
> some of these cat get localized (I remember having seen Martin work
> on something like this in the past).

I'd say a better statement is "rely only on error strings that you
control; don't rely on Python's internal error strings".
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From aahz@pythoncraft.com  Wed Dec 18 13:04:11 2002
From: aahz@pythoncraft.com (Aahz)
Date: Wed, 18 Dec 2002 08:04:11 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212170639.33487.mclay@nist.gov>
References: <LNBBLJKPBEHFEDALKOLCCEOADCAB.tim.one@comcast.net> <200212170639.33487.mclay@nist.gov>
Message-ID: <20021218130411.GA1503@panix.com>

On Tue, Dec 17, 2002, Michael McLay wrote:
> On Tuesday 17 December 2002 12:34 am, Tim Peters wrote:
>> [Michael McLay]
>>>
>>> To bad the decNumber library from IBM is not open source.
>>
>> A Java implementation is open:
>>
>>     http://www2.hursley.ibm.com/decimalj/
> 
> That's great for jython, but it will require more work for the C Python 
> implementation.

Much of a first draft of a Python implementation is available at my Web
site.  Tim's jab at me ("fifteen years") is because it really needs to
be rewritten from scratch and refactored, and I haven't gotten the
mental traction to do it.  Currently it does all the basics other than
division (which is when I realized that it needed rewriting).
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From guido@python.org  Wed Dec 18 13:26:21 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Dec 2002 08:26:21 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint) numbers to Python
In-Reply-To: Your message of "Tue, 17 Dec 2002 23:44:43 EST."
 <LNBBLJKPBEHFEDALKOLCAEPJDCAB.tim_one@email.msn.com>
References: <LNBBLJKPBEHFEDALKOLCAEPJDCAB.tim_one@email.msn.com>
Message-ID: <200212181326.gBIDQLS18363@pcp02138704pcs.reston01.va.comcast.net>

I've thought about this a bit, and I find fixedpoint or fixed or
fixpoint all much better names than decimal.  And fixed_point_decimal
(or variations thereof) is simply too long.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From mwh@python.net  Wed Dec 18 13:30:14 2002
From: mwh@python.net (Michael Hudson)
Date: 18 Dec 2002 13:30:14 +0000
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src setup.py,1.126,1.127
In-Reply-To: Andrew Kuchling's message of "Tue, 17 Dec 2002 12:17:57 -0500"
References: <E18OKsK-0002Mu-00@sc8-pr-cvs1.sourceforge.net> <20021217171757.GA16244@ute.mems-exchange.org>
Message-ID: <2mu1hbbgk9.fsf@starship.python.net>

Andrew Kuchling <akuchlin@mems-exchange.org> writes:

> On Tue, Dec 17, 2002 at 08:47:20AM -0800, mwh@users.sourceforge.net wrote:
> >+     # Do exactly what install_lib does but make sure correct access modes get
> >+     # set on installed directories and files. All installed files with get
> >+     # mode 644 unless they are a shared library in which case they will get
> >+     # mode 755. All installed directories will get mode 755.
> 
> If install_lib is getting the mode wrong, shouldn't this fix be applied to 
> install_lib instead of just the setup.py?

Probably.  Do you have the time/inclination to work out any possible
backwards compatibility interactions, where it needs to go, etc.?  If
you do, be my guest.  I don't, and neither did the guy who wrote the
patch.

Cheers,
M.

-- 
  Imagine if every Thursday your shoes exploded if you tied them
  the usual way.  This happens to us all the time with computers,
  and nobody thinks of complaining.                     -- Jeff Raskin


From guido@python.org  Wed Dec 18 13:33:23 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Dec 2002 08:33:23 -0500
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: Your message of "Wed, 18 Dec 2002 09:37:55 GMT."
 <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com>
References: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com>
Message-ID: <200212181333.gBIDXN518460@pcp02138704pcs.reston01.va.comcast.net>

> My specific issue is with get_data, which is totally unneeded
> for the import protocol itself (although it does solve real,
> related issues). I don't think it should be a requirement of
> the import hook API. Documenting it as an optional interface
> which hooks may choose to support may be reasonable, though.

I like get_data().  The caller should always be prepared for it to
return an error (how does it do that? raise what? or return None? that
should be specified!).  An importer that can't do this should
implement a version that always reports an error.  This is much easier
on the client than calling hasattr(importer, 'get_data') first or
catching AttributeError.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Wed Dec 18 13:34:18 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Dec 2002 08:34:18 -0500
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: Your message of "18 Dec 2002 11:25:16 +0100."
 <m3znr3lj3n.fsf@mira.informatik.hu-berlin.de>
References: <200212161609.03192.pobrien@orbtech.com> <200212170912.11638.pobrien@orbtech.com> <m3znr4haow.fsf@mira.informatik.hu-berlin.de> <200212171823.05716.pobrien@orbtech.com>
 <m3znr3lj3n.fsf@mira.informatik.hu-berlin.de>
Message-ID: <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net>

> It is, strictly speaking, a bug, since the documentation says that all
> objects will be recorded in the memo. The question is whether this is
> a documentation bug (i.e. the documentation is promising too much) or
> an implementation bug in cPickle.

Doc bug.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From fredrik@pythonware.com  Wed Dec 18 13:48:15 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Wed, 18 Dec 2002 14:48:15 +0100
Subject: [Python-Dev] Zipping Zope3
References: <16E1010E4581B049ABC51D4975CEDB88619989@UKDCX001.uk.int.atosorigin.com>  <200212181333.gBIDXN518460@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <02c501c2a69c$1e0135e0$0900a8c0@spiff>

Guido van Rossum wrote:

> I like get_data().  The caller should always be prepared for it to
> return an error (how does it do that? raise what? or return None? that
> should be specified!).  An importer that can't do this should
> implement a version that always reports an error.  This is much easier
> on the client than calling hasattr(importer, 'get_data') first or
> catching AttributeError.

IOError makes a certain kind of sense.

on the other hand, I'm not sure that requiring the use of hasattr/try-
except is that much of a pain, in practice.

:::

And if we're making get_data part of the protocol, can we please have
an optional get_module_names/list_modules/listdir method too?

Quoting myself from an earlier post (update the examples as necessary):

    as I might have mentioned before, it would be really nice if custom
    importers supported optional "load_data" and "get_module_names"
    hooks.

    the former can be used by code that looks for data files on the path
    (Zope, PIL, most likely hundreds of other tools/programs):

        for path in sys.path:
            if hasattr(path, "load_data"):
                try:
                    data =3D path.load_data(name)
                except IOError:
                    pass
                else:
                    return data # got it
            if isinstance(path, types.StringTypes) and =
os.path.isdir(path):
                try:
                    data =3D open(os.path.join(path, file), "rb").read()
                except IOError:
                    pass
                else:
                    return data
            # ...

    the latter can be used by code that looks for plugins using a
    name pattern (PIL's *ImagePlugins, for example):

        for path in sys.path:
            if hasattr(path, "get_module_names"):
                names =3D path.get_module_names()
            elif isinstance(path, types.StringTypes) and =
os.path.isdir(path):
                names =3D os.listdir(path)
            else:
                continue
            for name in names:
                if name.endswith("ImagePlugin"):
                    __import__(name)

</F>



From pedronis@bluewin.ch  Wed Dec 18 13:55:42 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Wed, 18 Dec 2002 14:55:42 +0100
Subject: [Python-Dev] Zipping Zope3
References: <r01050400-1022-969E3258126F11D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <004f01c2a69d$27f8faa0$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> > .... and I've just added a comment expressing some concerns.
>
> Likewise ;-)
>
> > I think we need a PEP about now, pinning down the "abstract"
> > import hook interface. What zipimport does risks turning into
> > a de-facto requirement, before we have any other hooks to
> > compare with.
>
> True.
>

I cannot do this for the first 2.3 alpha, but between it and the beta, I will
try to implement such a PEP for Jython in order not to underestimate some
details. One thing is that in Jython import logic we do not have the
find_module/load_module logic...

regards.



From just@letterror.com  Wed Dec 18 14:06:40 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 18 Dec 2002 15:06:40 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <004f01c2a69d$27f8faa0$6d94fea9@newmexico>
Message-ID: <r01050400-1022-04CDE07F129211D7A7C4003065D5E7E4@[10.0.0.1]>

Samuele Pedroni wrote:

> I cannot do this for the first 2.3 alpha, but between it and the
> beta, I will try to implement such a PEP for Jython in order not to
> underestimate some details. One thing is that in Jython import logic
> we do not have the find_module/load_module logic...

That's not neccesarily a showstopper for the importer protocol: during
an actual import Python calls find_module and load_module quickly after
one another, so this can be captured in an import_module function which
simply does:

    def import_module(fullname):
        loader = importer.find_module(fullname)
        if loader:
            return loader.load_module(fullname)
        return None  # raise ImportError, whatever

Paul Moore and I are now working on a PEP and we're making pretty good
progress. If you (or anyone for that matter) want to be involved
(passively or actively) before it gets posted, let me know.

Just


From Paul.Moore@atosorigin.com  Wed Dec 18 14:07:36 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 18 Dec 2002 14:07:36 -0000
Subject: [Python-Dev] Zipping Zope3
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861998C@UKDCX001.uk.int.atosorigin.com>

From: Guido van Rossum [mailto:guido@python.org]

>> My specific issue is with get_data, which is totally unneeded
>> for the import protocol itself (although it does solve real,
>> related issues). I don't think it should be a requirement of
>> the import hook API. Documenting it as an optional interface
>> which hooks may choose to support may be reasonable, though.

> I like get_data(). The caller should always be prepared for
> it to return an error (how does it do that? raise what? or
> return None? that should be specified!). An importer that
> can't do this should implement a version that always reports
> an error. This is much easier on the client than calling
> hasattr(importer, 'get_data') first or catching AttributeError.

I don't have a big problem with get_data(). But currently, it's
badly under-specified. You mention the question of how it returns
errors (not just "can't do this" - what about "file not found"?).
There's also the question of the exact format of the "filename".
Should it be like a module name (dotted)? This has the advantage
of matching how the rest of the importer logic works, but the
disadvantage of being unnatural to the user. Should it be a
pseudo-path format? This has the disadvantage of being platform
specific (the zipimporter already jumps through some hoops to
convert os.sep/os.altsep into forward slashes (which is what the
zip format uses), and other importers may have even more work to
do). Or something else?

Also is it a method of the object that implements load_module, or
the one that implements find_module? I could argue either. I can
code things either way. But I don't have enough "real world"
experience to say which is best in practice.

Paul.


From Paul.Moore@atosorigin.com  Wed Dec 18 15:05:02 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Wed, 18 Dec 2002 15:05:02 -0000
Subject: [Python-Dev] Zipping Zope3
Message-ID: <16E1010E4581B049ABC51D4975CEDB8861998D@UKDCX001.uk.int.atosorigin.com>

Fredrik Lundh said:
> And if we're making get_data part of the protocol, can we
> please have an optional get_module_names/list_modules/listdir
> method too?

This is why I'm concerned about get_data. It isn't necessarily
enough - we've already had people ask about get_module_names and
stat. Who knows what else could be useful? I suspect that some
things simply aren't going to be noticed until the new import
hooks get actually used "in the wild".

At the moment, this issue is in the "Open Issues" section of the
(not yet released) PEP.

Paul


From pedronis@bluewin.ch  Wed Dec 18 15:11:34 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Wed, 18 Dec 2002 16:11:34 +0100
Subject: [Python-Dev] Zipping Zope3
References: <r01050400-1022-04CDE07F129211D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <010301c2a6a7$c11dcc60$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> Paul Moore and I are now working on a PEP and we're making pretty good
> progress. If you (or anyone for that matter) want to be involved
> (passively or actively) before it gets posted, let me know.

Question: how does the patch respectively PEP features interact with imp
module?




From pobrien@orbtech.com  Wed Dec 18 15:41:30 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Wed, 18 Dec 2002 09:41:30 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net>
References: <200212161609.03192.pobrien@orbtech.com> <m3znr3lj3n.fsf@mira.informatik.hu-berlin.de> <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <200212180941.30127.pobrien@orbtech.com>

On Wednesday 18 December 2002 07:34 am, Guido van Rossum wrote:
> > It is, strictly speaking, a bug, since the documentation says that
> > all objects will be recorded in the memo. The question is whether
> > this is a documentation bug (i.e. the documentation is promising
> > too much) or an implementation bug in cPickle.
>
> Doc bug.

Not sure I completely agree, but I'm not trying to pick a fight, either. 
I just wish the following bit from the docs was true, whether pickle or 
cPickle were used, whether the object was first pickled inside some 
container and later pickled on its own, whether the instance was a 
new-style class or an old-style class, etc.:

<quote>
It is possible to make multiple calls to the dump() method of the same 
Pickler instance. These must then be matched to the same number of 
calls to the load() method of the corresponding Unpickler instance. If 
the same object is pickled by multiple dump() calls, the load() will 
all yield references to the same object.
</quote>

The fact that I'm no longer sure whether I'll get back a reference to a 
previously pickle instance or not calls into question whether pickling 
can be the foundation for a reliable Python persistence system. I can't 
help but express my disappointment. Was I really expecting too much 
from pickle/cPickle?

In any case, I've submitted the following documentation bug report on 
SF:

  [ 655802 ] cPickle not always same as pickle

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------



From just@letterror.com  Wed Dec 18 15:55:13 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 18 Dec 2002 16:55:13 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <010301c2a6a7$c11dcc60$6d94fea9@newmexico>
Message-ID: <r01050400-1022-1A167E5412A111D7A7C4003065D5E7E4@[10.0.0.1]>

Samuele Pedroni wrote:

> Question: how does the patch respectively PEP features interact with
> imp module?

Sneak preview from the PEP (glad I already wrote that part ;-):

"""
imp module integration

The new import hooks are not easily integrated in the existing
imp.find_module() and imp.load_module() calls. It's questionable whether
it's possible at all without breaking code; it is better to simply add a
new function to the imp module. The meaning of the existing
imp.find_module() and imp.load_module() calls changes from: "they expose
the builtin import mechanism" to "they expose the basic *unhooked*
builtin import mechanism". They simply won't invoke any import hooks. A
new imp module function is proposed under the name "find_module2", with
is used like the following pattern:

    loader = imp.find_module2(fullname, path)
    if loader is not None:
        loader.load_module(fullname)

In the case of a "basic" import, one the imp.find_module() function
would handle, the loader object would be a wrapper for the current
output of imp.find_module(), and loader.load_module() would call
imp.load_module() with that output.

Note that this wrapper is currently not yet implemented, although a
Python prototype exists in the test_importhooks.py script (the
ImpWrapper class).
"""

It needs to be fleshed out a bit more, but this is the basic idea.

Just


From guido@python.org  Wed Dec 18 16:00:42 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Dec 2002 11:00:42 -0500
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: Your message of "Wed, 18 Dec 2002 09:41:30 CST."
 <200212180941.30127.pobrien@orbtech.com>
References: <200212161609.03192.pobrien@orbtech.com> <m3znr3lj3n.fsf@mira.informatik.hu-berlin.de> <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net>
 <200212180941.30127.pobrien@orbtech.com>
Message-ID: <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net>

> Not sure I completely agree, but I'm not trying to pick a fight, either. 
> I just wish the following bit from the docs was true, whether pickle or 
> cPickle were used, whether the object was first pickled inside some 
> container and later pickled on its own, whether the instance was a 
> new-style class or an old-style class, etc.:
> 
> <quote>
> It is possible to make multiple calls to the dump() method of the same 
> Pickler instance. These must then be matched to the same number of 
> calls to the load() method of the corresponding Unpickler instance. If 
> the same object is pickled by multiple dump() calls, the load() will 
> all yield references to the same object.
> </quote>
> 
> The fact that I'm no longer sure whether I'll get back a reference to a 
> previously pickle instance or not calls into question whether pickling 
> can be the foundation for a reliable Python persistence system. I can't 
> help but express my disappointment. Was I really expecting too much 
> from pickle/cPickle?

Probably.  I have to admit that the "multiple calls to dump()" feature
is not often used; maybe it should be deprecated so we don't make
promises we can't fulfill.  E.g. Zope never uses this.

> In any case, I've submitted the following documentation bug report on 
> SF:
> 
>   [ 655802 ] cPickle not always same as pickle

You could help by submitting a doc patch that satisfies your
desire for more clear and honest docs.  If you don't know LaTeX, that
doesn't matter; just pretend it's plain text and somebody will fix up
the markup, as long as you provide the new words.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pobrien@orbtech.com  Wed Dec 18 16:22:52 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Wed, 18 Dec 2002 10:22:52 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net>
References: <200212161609.03192.pobrien@orbtech.com> <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <200212181022.52849.pobrien@orbtech.com>

On Wednesday 18 December 2002 10:00 am, Guido van Rossum wrote:
> I have to admit that the "multiple calls to dump()" feature
> is not often used; maybe it should be deprecated so we don't make
> promises we can't fulfill.  E.g. Zope never uses this.

Since it works 99% of the time, I'd rather fix it and make it work 100% 
of the time. If it can't be fixed, the promises made in the docs should 
be tempered. See my next suggestion first.

Here is another thought that might outway the performance concerns: for 
whatever reason, all instances of new-style classes (even with refcount 
== 1, or perhaps because they don't have a refcount == 1) *are* getting 
put in the cPickle memo. If the goal is to move from old-style to 
new-style, we're already taking a performace hit with new-style class 
instances. So why not pay the price with old-style as well and put them 
in the cPickle memo? Then the discrepancy between pickle and cPickle 
goes away, and this "multiple dump" feature works as advertised.

Or create a second version of cPickle that does work exactly the same as 
pickle. It will still have a performance benefit over pickle and it 
won't be any slower than the regular cPickle if all your instances are 
of new-style classes. Just thinking out loud.

> > In any case, I've submitted the following documentation bug report
> > on SF:
> >
> >   [ 655802 ] cPickle not always same as pickle
>
> You could help by submitting a doc patch that satisfies your
> desire for more clear and honest docs.  If you don't know LaTeX, that
> doesn't matter; just pretend it's plain text and somebody will fix up
> the markup, as long as you provide the new words.

I'll add this to my todo list, assuming you don't like my suggestions 
above. My documentation bug report contained annotations that suggest 
some wording changes already, but I'm sure Fred would prefer patches 
with specific revisions.

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------



From jim@zope.com  Wed Dec 18 18:36:49 2002
From: jim@zope.com (Jim Fulton)
Date: Wed, 18 Dec 2002 13:36:49 -0500
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
References: <200212161609.03192.pobrien@orbtech.com> <m3znr3lj3n.fsf@mira.informatik.hu-berlin.de> <200212181334.gBIDYIB18483@pcp02138704pcs.reston01.va.comcast.net>              <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <3E00C041.7020901@zope.com>

Guido van Rossum wrote:
>>Not sure I completely agree, but I'm not trying to pick a fight, either. 
>>I just wish the following bit from the docs was true, whether pickle or 
>>cPickle were used, whether the object was first pickled inside some 
>>container and later pickled on its own, whether the instance was a 
>>new-style class or an old-style class, etc.:
>>
>><quote>
>>It is possible to make multiple calls to the dump() method of the same 
>>Pickler instance. These must then be matched to the same number of 
>>calls to the load() method of the corresponding Unpickler instance. If 
>>the same object is pickled by multiple dump() calls, the load() will 
>>all yield references to the same object.
>></quote>
>>
>>The fact that I'm no longer sure whether I'll get back a reference to a 
>>previously pickle instance or not calls into question whether pickling 
>>can be the foundation for a reliable Python persistence system. I can't 
>>help but express my disappointment. Was I really expecting too much 
>>from pickle/cPickle?
>>
> 
> Probably.  I have to admit that the "multiple calls to dump()" feature
> is not often used; maybe it should be deprecated so we don't make
> promises we can't fulfill.  E.g. Zope never uses this.

Zope does use this.

Zope doesn't pickle the same object multiple times in the separate
load calls, however.  This may be a matter of luck.

I think Patrick raises a reasonable issue, although it would require
slowing things down a bit to fix it.

Patrick, why do you feel you need this for a reliable persistence
system? ZODB doesn't need this.

Jim


-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (888) 344-4332            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org



From pobrien@orbtech.com  Wed Dec 18 19:28:07 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Wed, 18 Dec 2002 13:28:07 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <3E00C041.7020901@zope.com>
References: <200212161609.03192.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net> <3E00C041.7020901@zope.com>
Message-ID: <200212181328.07455.pobrien@orbtech.com>

On Wednesday 18 December 2002 12:36 pm, Jim Fulton wrote:
> I think Patrick raises a reasonable issue, although it would require
> slowing things down a bit to fix it.
>
> Patrick, why do you feel you need this for a reliable persistence
> system? ZODB doesn't need this.

I don't necessarily *need* this, but it would be nice to have. Otherwise 
I have to remember to always work around it. Here is a quick rundown on 
my particular use case:

I'm working on a very simple persistence framework, called PyPerSyst 
(http://sourceforge.net/projects/pypersyst), whose goal is to be able 
to persist any picklable object graph, support ACID requirements, and 
be able to safely recover from crashes. The difference between 
PyPerSyst and ZODB is that PyPerSyst keeps everything in RAM (which is 
how it keeps things simple).

The basic approach is to start by pickling an object graph (which we 
call the System). Changes to the System are then made by sending a 
Transaction class instance to the persistence Engine (sort of a 
mediator thing). The Engine pickles the Transaction to a transaction 
log, then executes the transaction against the System. If there is a 
crash, the System is recovered from the pickle file and then all 
Transactions in the log are loaded and re-executed against the System.

The actual pickling strategy is delegated by the Engine to one of 
several Storage classes. One of the storages uses a single file (look 
for `singlefile.py`) for the System snapshot as well as all 
Transactions. This storage also allows Transactions to contain direct 
references to objects in the System.

In order for this to work properly, the object references in the 
Transactions must resolve to the same objects in the System when 
Transactions are unpickled during recovery. In our first sample app (a 
Banking app, no less) this didn't work properly because of the "bug" in 
cPickle (and recent bank account transactions would effectively 
disappear, even though they appeared to be recovered properly).

Because this is a framework that is supposed to work with any System and 
any Transaction defined by the end-user, I can't easily guarantee that 
they won't be referencing class instances that cPickle won't want to 
memoize. So I either need to disallow this particular storage strategy, 
or have it use only pickle (though I'd like the speed of cPickle), or 
make sure cPickle works the same as pickle.

Did any of this make sense? Did I answer your question?

[The other thing I don't know is if this "bug" that we are discussing is 
the only example of its kind, or whether there are more to be 
discovered. But we can cross that bridge when we get there.]

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------



From guido@python.org  Wed Dec 18 19:39:32 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Dec 2002 14:39:32 -0500
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: Your message of "Wed, 18 Dec 2002 10:22:52 CST."
 <200212181022.52849.pobrien@orbtech.com>
References: <200212161609.03192.pobrien@orbtech.com> <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net>
 <200212181022.52849.pobrien@orbtech.com>
Message-ID: <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net>

> Here is another thought that might outway the performance concerns: for 
> whatever reason, all instances of new-style classes (even with refcount 
> == 1, or perhaps because they don't have a refcount == 1) *are* getting 
> put in the cPickle memo.

Pickling of new-style classes is currently done using a Python helper.
I am still hoping that I'll be able to dig myself out of the pile of
Zope work that I'm currently under, and then I'd like to work on
making pickling of new-style classes a lot more efficient.  So please
don't use this argument to cripple pickle any further.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pobrien@orbtech.com  Wed Dec 18 19:44:43 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Wed, 18 Dec 2002 13:44:43 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net>
References: <200212161609.03192.pobrien@orbtech.com> <200212181022.52849.pobrien@orbtech.com> <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <200212181344.43536.pobrien@orbtech.com>

On Wednesday 18 December 2002 01:39 pm, Guido van Rossum wrote:
> Pickling of new-style classes is currently done using a Python
> helper. I am still hoping that I'll be able to dig myself out of the
> pile of Zope work that I'm currently under, and then I'd like to work
> on making pickling of new-style classes a lot more efficient.  So
> please don't use this argument to cripple pickle any further.

Okay. (ROFL) :-)

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------



From sholden@holdenweb.com  Wed Dec 18 20:17:51 2002
From: sholden@holdenweb.com (Steve Holden)
Date: Wed, 18 Dec 2002 15:17:51 -0500
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
References: <200212161609.03192.pobrien@orbtech.com> <200212180941.30127.pobrien@orbtech.com> <200212181600.gBIG0h419136@pcp02138704pcs.reston01.va.comcast.net>              <200212181022.52849.pobrien@orbtech.com>  <200212181939.gBIJdWV21046@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <014b01c2a6d2$8ba255d0$6b01a8c0@holdenweb.com>

> > Here is another thought that might outway the performance concerns: for
> > whatever reason, all instances of new-style classes (even with refcount
> > == 1, or perhaps because they don't have a refcount == 1) *are* getting
> > put in the cPickle memo.
>
> Pickling of new-style classes is currently done using a Python helper.
> I am still hoping that I'll be able to dig myself out of the pile of
> Zope work that I'm currently under, and then I'd like to work on
> making pickling of new-style classes a lot more efficient.  So please
> don't use this argument to cripple pickle any further.
>

I'm not a crippled pickle, I'm a crippled pickle's son
and I'll be finished crippling pickles when the pickle crippling's done.

Sorry. Been working too hard ...
-----------------------------------------------------------------------
Steve Holden                                  http://www.holdenweb.com/
Python Web Programming                 http://pydish.holdenweb.com/pwp/
Bring your musical instrument to PyCon!    http://www.python.org/pycon/
-----------------------------------------------------------------------





From greg@cosc.canterbury.ac.nz  Wed Dec 18 20:53:15 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Dec 2002 09:53:15 +1300 (NZDT)
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEPJDCAB.tim_one@email.msn.com>
Message-ID: <200212182053.gBIKrFb27247@oma.cosc.canterbury.ac.nz>

Tim Peters <tim_one@email.msn.com>:

> So if the fixed decimal type is called "decimal", what are *you* going to
> call the new floating decimal type?

I don't know, but whatever it is it somehow needs to connote
both decimalness and floatness.

Maybe the module should be called decimal, then we could
have decimal.fixed and decimal.float.

Or maybe decimal could be a built-in abstract type with
class attributes fixed and float that are subclasses of
it.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From Jack.Jansen@oratrix.com  Wed Dec 18 21:50:02 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Wed, 18 Dec 2002 22:50:02 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <r01050400-1022-1A167E5412A111D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <A9433EC6-12D2-11D7-A2C1-000A27B19B96@oratrix.com>

On woensdag, dec 18, 2002, at 16:55 Europe/Amsterdam, Just van Rossum 
wrote:
> The new import hooks are not easily integrated in the existing
> imp.find_module() and imp.load_module() calls. It's questionable 
> whether
> it's possible at all without breaking code; it is better to simply add 
> a
> new function to the imp module. The meaning of the existing
> imp.find_module() and imp.load_module() calls changes from: "they 
> expose
> the builtin import mechanism" to "they expose the basic *unhooked*
> builtin import mechanism". They simply won't invoke any import hooks. A
> new imp module function is proposed under the name "find_module2", with
> is used like the following pattern:
>
>     loader = imp.find_module2(fullname, path)
>     if loader is not None:
>         loader.load_module(fullname)

If we want to use this for freeze-like functionality then the loader 
will also need a method (or methods) that at the very least return (for 
modules for which it makes sense) either the source code for the module 
or a file-like object that can be used to read the source code.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From greg@cosc.canterbury.ac.nz  Wed Dec 18 21:45:00 2002
From: greg@cosc.canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Dec 2002 10:45:00 +1300 (NZDT)
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212181022.52849.pobrien@orbtech.com>
Message-ID: <200212182145.gBILj0E27411@oma.cosc.canterbury.ac.nz>

"Patrick K. O'Brien" <pobrien@orbtech.com>:

> Or create a second version of cPickle that does work exactly the same as 
> pickle.

Or make it an option or alternative entry point to the
existing cPickle. The two cPickles would be almost
exactly the same code-wise, wouldn't they?

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+


From just@letterror.com  Wed Dec 18 22:34:20 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 18 Dec 2002 23:34:20 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <A9433EC6-12D2-11D7-A2C1-000A27B19B96@oratrix.com>
Message-ID: <r01050400-1022-E00BBF9F12D811D7A7C4003065D5E7E4@[10.0.0.1]>

Jack Jansen wrote:

> >     loader = imp.find_module2(fullname, path)
> >     if loader is not None:
> >         loader.load_module(fullname)
> 
> If we want to use this for freeze-like functionality then the loader
> will also need a method (or methods) that at the very least return
> (for modules for which it makes sense) either the source code for the
> module or a file-like object that can be used to read the source code.

Features schmeatures! I would agree, but Paul and I have decided to keep
the importer protocol real simple, at least for now. Extensions to the
importer protocol are definitely needed, but we'd like to concentrate on
the basics first. We (as in: all of us) simply need more time to come up
with a decent, coherent and simple set of (optional) extensions to the
protocol. The zipimporter object now has a get_data(filename) method
with which you can do what you want (in a less convenient way), and Paul
questions even that addition, as we might not be able to go back once
it's in the core. It could even be the subject of a PEP of its own ;-)

Also: *anything* that makes zipimporter closer to a file system is going
at it from the wrong direction. We *first* need a virtual file system,
then an import hook that can import from it, and *then* a zip back
end...

Just


From pobrien@orbtech.com  Wed Dec 18 23:25:29 2002
From: pobrien@orbtech.com (Patrick K. O'Brien)
Date: Wed, 18 Dec 2002 17:25:29 -0600
Subject: [Python-Dev] [ 654866 ] pickle and cPickle not equivalent
In-Reply-To: <200212182145.gBILj0E27411@oma.cosc.canterbury.ac.nz>
References: <200212182145.gBILj0E27411@oma.cosc.canterbury.ac.nz>
Message-ID: <200212181725.29937.pobrien@orbtech.com>

On Wednesday 18 December 2002 03:45 pm, Greg Ewing wrote:
> "Patrick K. O'Brien" <pobrien@orbtech.com>:
> > Or create a second version of cPickle that does work exactly the
> > same as pickle.
>
> Or make it an option or alternative entry point to the
> existing cPickle. The two cPickles would be almost
> exactly the same code-wise, wouldn't they?

I agree. I was speaking in general terms. The implementation should 
avoid duplicating code.

-- 
Patrick K. O'Brien
Orbtech      http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------



From neal@metaslash.com  Thu Dec 19 00:44:45 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Wed, 18 Dec 2002 19:44:45 -0500
Subject: [Python-Dev] test_datetime fails on alpha
Message-ID: <20021219004445.GD12063@epoch.metaslash.com>

I used the alpha in the SF compile to verify the bug.  The problem was
originally reported on the snake-farm from the test suite.

nnorwitz@usf-cf-alpha-linux-1:~/python/dist/src$ ./python 
Python 2.3a0 (#15, Dec 18 2002, 15:52:53) 
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import timedelta
>>> timedelta(1, 7, 31)
datetime.timedelta(1, 7, 31)
>>> _.days, _.seconds, _.microseconds
(30064771073, 133143986183, 31)

Neal


From tim.one@comcast.net  Thu Dec 19 01:04:40 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Dec 2002 20:04:40 -0500
Subject: [Python-Dev] test_datetime fails on alpha
In-Reply-To: <20021219004445.GD12063@epoch.metaslash.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEEDDDAB.tim.one@comcast.net>

[Neal Norwitz]
> I used the alpha in the SF compile to verify the bug.  The problem was
> originally reported on the snake-farm from the test suite.
>
> nnorwitz@usf-cf-alpha-linux-1:~/python/dist/src$ ./python
> Python 2.3a0 (#15, Dec 18 2002, 15:52:53)
> [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from datetime import timedelta
> >>> timedelta(1, 7, 31)
> datetime.timedelta(1, 7, 31)
> >>> _.days, _.seconds, _.microseconds
> (30064771073, 133143986183, 31)

It's really no help.  The *first* thing to do is to build Python with
optimzation disabled.  Try a debug build, if possible, and step thru
delta_new().  Luckily, that's the hairiest code in the whole module <wink>.

Putting the "wrong parts" in hex, that result is

(0x700000001L, 0x1F00000007L, 31)

which suggests nothing to me.  I don't see any 32-bit assumptions in the
code here, and thanks to two previous lives on 64-bit boxes they usually
punch me in the eyeball if they exist.



From jepler@unpythonic.net  Thu Dec 19 00:30:34 2002
From: jepler@unpythonic.net (Jeff Epler)
Date: Wed, 18 Dec 2002 18:30:34 -0600
Subject: [Python-Dev] test_datetime fails on alpha
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEEDDDAB.tim.one@comcast.net>
References: <20021219004445.GD12063@epoch.metaslash.com> <LNBBLJKPBEHFEDALKOLCEEEDDDAB.tim.one@comcast.net>
Message-ID: <20021218183022.A987@unpythonic.net>

On Wed, Dec 18, 2002 at 08:04:40PM -0500, Tim Peters wrote:
> [Neal Norwitz]
> > I used the alpha in the SF compile to verify the bug.  The problem was
> > originally reported on the snake-farm from the test suite.
> >
> > nnorwitz@usf-cf-alpha-linux-1:~/python/dist/src$ ./python
> > Python 2.3a0 (#15, Dec 18 2002, 15:52:53)
> > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> from datetime import timedelta
> > >>> timedelta(1, 7, 31)
> > datetime.timedelta(1, 7, 31)
> > >>> _.days, _.seconds, _.microseconds
> > (30064771073, 133143986183, 31)
> 
> It's really no help.  The *first* thing to do is to build Python with
> optimzation disabled.  Try a debug build, if possible, and step thru
> delta_new().  Luckily, that's the hairiest code in the whole module <wink>.
> 
> Putting the "wrong parts" in hex, that result is
> 
> (0x700000001L, 0x1F00000007L, 31)
     ^                      ^  (0x1F)
                   ^^           ^^
> which suggests nothing to me.

something is sure suggested to me by those numbers.  It's as though
successive members are 4 bytes apart, but they're being treated as 8 bytes
wide, or something.

how the code manages it is beyond me, though.  (I don't even see a
"datetime" module in my CVS tree)

Jeff


From neal@metaslash.com  Thu Dec 19 02:34:22 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Wed, 18 Dec 2002 21:34:22 -0500
Subject: [Python-Dev] test_datetime fails on alpha
In-Reply-To: <20021218183022.A987@unpythonic.net>
References: <20021219004445.GD12063@epoch.metaslash.com> <LNBBLJKPBEHFEDALKOLCEEEDDDAB.tim.one@comcast.net> <20021218183022.A987@unpythonic.net>
Message-ID: <20021219023422.GE12063@epoch.metaslash.com>

On Wed, Dec 18, 2002 at 06:30:34PM -0600, Jeff Epler wrote:
> On Wed, Dec 18, 2002 at 08:04:40PM -0500, Tim Peters wrote:
> > [Neal Norwitz]
> > > I used the alpha in the SF compile to verify the bug.  The problem was
> > > originally reported on the snake-farm from the test suite.
> > >
> > > >>> from datetime import timedelta
> > > >>> timedelta(1, 7, 31)
> > > datetime.timedelta(1, 7, 31)
> > > >>> _.days, _.seconds, _.microseconds
> > > (30064771073, 133143986183, 31)
> > 
> > It's really no help.  The *first* thing to do is to build Python with
> > optimzation disabled.  Try a debug build, if possible, and step thru
> > delta_new().  Luckily, that's the hairiest code in the whole module <wink>.
> > 
> > Putting the "wrong parts" in hex, that result is
> > 
> > (0x700000001L, 0x1F00000007L, 31)
>      ^                      ^  (0x1F)
>                    ^^           ^^
> > which suggests nothing to me.
> 
> something is sure suggested to me by those numbers.  It's as though
> successive members are 4 bytes apart, but they're being treated as 8 bytes
> wide, or something.

That kept bugging me.  I just checked in the fix.  The problem was the
accessors members for days, seconds, and microseconds were T_LONGs
instead of T_INTs.

Test passes on alpha & x86 for me.

Neal



From guido@python.org  Thu Dec 19 03:54:34 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Dec 2002 22:54:34 -0500
Subject: [Python-Dev] deprecating APIs
In-Reply-To: Your message of "Mon, 16 Dec 2002 20:49:35 EST."
 <20021217014935.GQ16807@epoch.metaslash.com>
References: <20021217014935.GQ16807@epoch.metaslash.com>
Message-ID: <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net>

> Are we still planning to provide a mechanism to declare APIs, etc as
> deprecated?

I think this is a neat idea.

> I have this code added to pyport.h which works for gcc:
> 
>     /* Py_DEPRECATED(version)
>      * Declare a macro or function deprecated.
>      * Usage:
>      *    extern int old_var Py_DEPRECATED(2.3);
>      *    typedef int T1 Py_DEPRECATED(2.4);
>      *    extern int x() Py_DEPRECATED(2.5);
>      */
>     #if defined(__GNUC__) && (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1)
>     #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
>     #else
>     #define Py_DEPRECATED(VERSION_UNUSED) 
>     #endif
> 
> The version number is not necessary.  It is only for documentation.
> I don't know how to support other compilers.

What does this do under GCC?  I suppose it issues a warning when you
use the deprecated API?

> We could add Py_USE_DEPRECATED and/or Py_HIDE_DEPRECATED for
> providing/hiding access to deprecated features.
> 
>     #ifdef Py_USE_DEPRECATED
>     #define Py_SOME_DEPRECATED_MACRO
>     #endif
> 
> or
> 
>     #ifndef Py_HIDE_DEPRECATED
>     #define Py_SOME_DEPRECATED_MACRO
>     #endif
> 
> Unfortunately, the Py_DEPRECATED macro doesn't work with macros, so we
> would have to use Py_USE_DEPRECATED/Py_HIDE_DEPRECATED around
> deprecated macros.

That's OK.  Just the fact that when reading the .h files one sees the
DEPRECATED keyword should be a big hint...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@comcast.net  Thu Dec 19 05:58:37 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Dec 2002 00:58:37 -0500
Subject: [Python-Dev] RE: Adding decimal (aka FixedPoint)  numbers to Python
In-Reply-To: <200212170639.33487.mclay@nist.gov>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEEODDAB.tim.one@comcast.net>

[Michael McLay]
> ...
> be a reasonable start. I am concerned about one thing. Will the
> fixed point implementation you have implemented be backwards
> compatible with an implementation of Cowlishaw's decimal numbers?

Nope.  Any scheme that can lose information is, in general, incompatible
with any other scheme that can lose information, unless they lose
information in identical ways in all cases.  Since Cowlishaw's is a floating
scheme, and FixedPoint is a fixed scheme, they'll get different results in
some information-losing cases.

Examples aren't hard to find.  Here's one:  what's 13% of 100006.73?

The true result is exactly 10000673 * 13 / 10000 = 13000.8749.

Do that via fixed-points with 2 digits after the decimal point, and it's
rounded down to 13000.87 (which is the best possible result if you get to
keep no more than two digits after the decimal point).

Use Cowlishaw's scheme with 8 digits of precision (which counts all
digits -- the decimal point floats) instead, and the trailing 9 has to be
tossed, and so rounds the last retained digit from 4 up to 5:  13000.875.
You've got 8 digits now, but a worse result if pennies are what you need:
if you round *that* to the nearest penny, to-nearest/even ("banker's
rounding") says "OK, I have to toss the 5, and 7 is odd, so round it up the
even 8", leaving 13000.88.

The end results aren't the same, and I recall now that this is why
FixedPoint was designed to lose info on multiply:  when you're computing
percentages to the nearest penny (be it sales tax or rebates), you can't get
the best possible result in all cases if you use a scheme that rounds twice.
A fixed-point scheme can get away with rounding just once (& very
naturally), but a floating scheme can have "too many digits", and therefore
force the user to do a second rounding step to cut the result back to the
number of fractional digits they wanted.  Double-rounding can't be as good
as single-rounding in all cases, unless the floating scheme uses much more
precision than the fixed scheme (there are tricky theorems that kick in
then -- if you have "enough" excess precision, it's provable that
double-rounding is always the same as single-rounding -- but not one user in
a million can be expected to know this, and even if they did the application
is too difficult since it effectively requires proving that you've always
got enough extra precision relative to your app's specific numbers).

BTW, this is why FixedPoint's "good to the last penny" has more to do with
the fixed-point business than with the decimal business.  Decimal arithmetic
is just as prone to double-rounding errors as binary arithmetic, and
floats-- whether decimal or binary --force the user into the double-rounding
business when they're working with fixed-point quantitities.



From fredrik@pythonware.com  Thu Dec 19 07:16:29 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Thu, 19 Dec 2002 08:16:29 +0100
Subject: [Python-Dev] Zipping Zope3
References: <r01050400-1022-E00BBF9F12D811D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <005201c2a72e$8ec4c6b0$ced241d5@hagrid>

Just van Rossum wrote:

> We (as in: all of us) simply need more time to come up with a decent,
> coherent and simple set of (optional) extensions to the protocol.

some of us has been shipping deployment tools for many years, and
have a pretty good understanding of what optional extensions we and
our customers have needed over the years...

</F>



From just@letterror.com  Thu Dec 19 09:17:33 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 19 Dec 2002 10:17:33 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <005201c2a72e$8ec4c6b0$ced241d5@hagrid>
Message-ID: <r01050400-1022-B8BE21B0133211D7A7C4003065D5E7E4@[10.0.0.1]>

Fredrik Lundh wrote:

> Just van Rossum wrote:
> 
> > We (as in: all of us) simply need more time to come up with a
> > decent, coherent and simple set of (optional) extensions to the
> > protocol.
> 
> some of us has been shipping deployment tools for many years, and
> have a pretty good understanding of what optional extensions we and
> our customers have needed over the years...

Ok, replace "all of us" by "all of us except /F".

Seriously, what do you want me to do? Guido asked for

    i.stat_file(filename)
    i.open_file(filename)

you asked for

    i.get_module_names()
    i.load_data()

Jack asked for

    i.get_module_code(modulename)
    i.get_module_source(modulename)

I'm sure someone suggested i.listdir() or else someone will.

Although I agree with the intentions of all of those suggestions,
there's just no way we can please everyone right now.

Btw. what you want is _possible_ with the  current zipimporter object:
there's zi.get_data(filename), and you can inspect the "files" attribute
(a dict mapping filenames to index info; it's an internal attribute, so
I might rename it to _files).

I have enough on my hands to get the PEP done and to get the
implementation and the basic design right. Design descisions about
(optional) extensions to the importer protocol will have to wait. If
someone wants to pick that up and make a concrete proposal, be my guest.

Just


From Paul.Moore@atosorigin.com  Thu Dec 19 09:22:37 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Thu, 19 Dec 2002 09:22:37 -0000
Subject: [Python-Dev] Zipping Zope3
Message-ID: <16E1010E4581B049ABC51D4975CEDB88619991@UKDCX001.uk.int.atosorigin.com>

From: Just van Rossum [mailto:just@letterror.com]
> Btw. what you want is _possible_ with the current zipimporter
> object: there's zi.get_data(filename), and you can inspect the
> "files" attribute (a dict mapping filenames to index info; it's
> an internal attribute, so I might rename it to _files).

Just to clarify this point: you can subclass the existing
zipimport.zipimporter class and add any extra methods you want.
Then, by replacing zipimporter in sys.path_hooks with your
subclass, you transparently get your extra functionality.

I'm not saying that this is the ultimate solution, but it gives
us an opportunity to experiment before committing to a specific
implementation in the core.

Paul.


From just@letterror.com  Thu Dec 19 09:34:35 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 19 Dec 2002 10:34:35 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88619991@UKDCX001.uk.int.atosorigin.com>
Message-ID: <r01050400-1022-1C0EC34C133511D7A7C4003065D5E7E4@[10.0.0.1]>

Moore, Paul wrote:

> Just to clarify this point: you can subclass the existing
> zipimport.zipimporter class and add any extra methods you want.
> Then, by replacing zipimporter in sys.path_hooks with your
> subclass, you transparently get your extra functionality.

It think would be better (for now) to just use the zipimporter object as
is, and write a helper function. Subclassing should be possible, but
might give you bootstrapping problems.

Just


From Jack.Jansen@oratrix.com  Thu Dec 19 12:56:49 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Thu, 19 Dec 2002 13:56:49 +0100
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: <r01050400-1022-E00BBF9F12D811D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <564D985C-1351-11D7-84BC-003065517236@oratrix.com>

On woensdag, dec 18, 2002, at 23:34 Europe/Amsterdam, Just van Rossum 
wrote:
> Features schmeatures! I would agree, but Paul and I have decided to 
> keep
> the importer protocol real simple, at least for now.

Agreed.
>  The zipimporter object now has a get_data(filename) method
> with which you can do what you want (in a less convenient way), and 
> Paul
> questions even that addition, as we might not be able to go back once
> it's in the core. It could even be the subject of a PEP of its own ;-)
>
> Also: *anything* that makes zipimporter closer to a file system is 
> going
> at it from the wrong direction. We *first* need a virtual file system,
> then an import hook that can import from it, and *then* a zip back
> end...

Ok, point taken,
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From just@letterror.com  Thu Dec 19 13:14:56 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 19 Dec 2002 14:14:56 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
Message-ID: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>

PEP: XXX
Title: New Import Hooks
Version: $Revision:$
Last-Modified: $Date:$
Author: Just van Rossum <just@letterror.com>,
    Paul Moore <gustav@morpheus.demon.co.uk>
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created:
Python-Version: 2.3
Post-History:


Abstract

    This PEP proposes to add a new set of import hooks that offer better
    customization of the Python import mechanism.  Contrary to the
    current __import__ hook, a new-style hook can be injected into the
    existing scheme, allowing for a finer grained control of how modules
    are found and how they are loaded.


Motivation

    The only way to customize the import mechanism is currently to
    override the builtin __import__ function.  However, overriding
    __import__ has many problems.  To begin with:

    - An __import__ replacement needs to *fully* reimplement the entire
      import mechanism, or call the original __import__ before or after
      the custom code.

    - It has very complex semantics and responsibilities.

    - __import__ gets called even for modules that are already in
      sys.modules, which is almost never what you want, unless you're
      writing some sort of monitoring tool.

    The situation gets worse when you need to extend the import
    mechanism from C: it's currently impossible, apart from hacking
    Python's import.c or reimplementing much of import.c from scratch.

    There is a fairly long history of tools written in Python that allow
    extending the import mechanism in various way, based on the
    __import__ hook.  The Standard Library includes two such tools:
    ihooks.py (by GvR) and imputil.py (Greg Stein), but perhaps the most
    famous is iu.py by Gordon McMillan, available as part of his
    Installer [1] package.  Their usefulness is somewhat limited because
    they are written in Python; bootstrapping issues need to worked
    around as you can't load the module containing the hook with the
    hook itself.  So if you want the entire Standard Library to be
    loadable from an import hook, the hook must be written in C.


Use cases

    This section lists several existing applications that depend on
    import hooks.  Among these, a lot of duplicate work was done that
    could have been saved if there had been a more flexible import hook
    at the time.  This PEP should make life a lot easier for similar
    projects in the future.

    Extending the import mechanism is needed when you want to load
    modules that are stored in a non-standard way.  Examples include
    modules that are bundled together in an archive; byte code that is
    not stored in a pyc formatted file; modules that are loaded from a
    database over a network.

    The work on this PEP was partly triggered by the implementation of
    PEP 273 [2], which adds imports from Zip archives as a builtin
    feature to Python.  While the PEP itself was widely accepted as a
    must-have feature, the implementation left a few things to desire.
    For one thing it went through great lengths to integrate itself with
    import.c, adding lots of code that was either specific for Zip file
    imports or *not* specific to Zip imports, yet was not generally
    useful (or even desirable) either.  Yet the PEP 273 implementation
    can hardly be blamed for this: it is simply extremely hard to do,
    given the current state of import.c.

    Packaging applications for end users is a typical use case for
    import hooks, if not *the* typical use case.  Distributing lots of
    source or pyc files around is not always appropriate (let alone a
    separate Python installation), so there is a frequent desire to
    package all needed modules in a single file.  So frequent in fact
    that multiple solutions have been implemented over the years.

    The oldest one is included with the Python source code: Freeze [3].
    It puts marshalled byte code into static objects in C source code.
    Freeze's "import hook" is hard wired into import.c, and has a couple
    of issues.  Later solutions include Fredrik Lundh's Squeeze [4],
    Gordon McMillan's Installer [1] and Thomas Heller's py2exe [5].
    MacPython ships with a tool called BuildApplication.

    Squeeze, Installer and py2exe use an __import__ based scheme (py2exe
    currently uses Installer's iu.py, Squeeze used ihooks.py), MacPython
    has two Mac-specific import hooks hard wired into import.c, that are
    similar to the Freeze hook.  The hooks proposed in this PEP enables
    us (at least in theory; it's not a short term goal) to get rid of
    the hard coded hooks in import.c, and would allow the
    __import__-based tools to get rid of most of their import.c
    emulation code.

    Before work on the design and implementation of this PEP was
    started, a new BuildApplication-like tool for MacOSX prompted one of
    the authors of this PEP (JvR) to expose the table of frozen modules
    to Python, in the imp module.  The main reason was to be able to use
    the freeze import hook (avoiding fancy __import__ support), yet to
    also be able to supply a set of modules at runtime.  This resulted
    in sf patch #642578 [6], which was mysteriously accepted (mostly
    because nobody seemed to care either way ;-).  Yet it is completely
    superfluous when this PEP gets accepted, as it offers a much nicer
    and general way to do the same thing.


Rationale

    While experimenting with alternative implementation ideas to get
    builtin Zip import, it was discovered that achieving this is
    possible with only a fairly small amount of changes to import.c.
    This allowed to factor out the Zip-specific stuff into a new source
    file, while at the same time creating a *general* new import hook
    scheme: the one you're reading about now.

    An earlier design allowed non-string objects on sys.path.  Such an
    object would have the neccesary methods to handle an import.  This
    has two disadvantages: 1) it breaks code that assumes all items on
    sys.path are strings; 2) it is not compatible with the PYTHONPATH
    environment variable.  The latter is directly needed for Zip
    imports.  A compromise came from Jython: allow string *subclasses*
    on sys.path, which would then act as importer objects.  This avoids
    some breakage, and seems to work well for Jython (where it is used
    to load modules from .jar files), but it was perceived as an "ugly
    hack".

    This lead to a more elaborate scheme, (mostly copied from McMillan's
    iu.py) in which each in a list of candidates is asked whether it can
    handle the sys.path item, until one is found that can.  This list of
    candidates is a new object in the sys module: sys.path_hooks.

    Traversing sys.path_hooks for each path item for each new import can
    be expensive, so the results are cached in another new object in the
    sys module: sys.path_importer_cache.  It maps sys.path entries to
    importer objects.

    To minimize the impact on import.c as well as to avoid adding extra
    overhead, it was chosen to not add an explicit hook and importer
    object for the existing file system import logic (as iu.py has), but
    to simply fall back to the builtin logic if no hook on
    sys.path_hooks could handle the path item.  If this is the case, a
    None value is stored in sys.path_importer_cache, again to avoid
    repeated lookups.  (Later we can go further and add a real importer
    object for the builtin mechanism, for now, the None fallback scheme
    should suffice.)

    A question was raised: what about importers that don't need *any*
    entry on sys.path? (Builtin and frozen modules fall into that
    category.) Again, Gordon McMillan to the rescue: iu.py contains a
    thing he calls the "metapath".  In this PEP's implementation, it's a
    list of importer objects that is traversed *before* sys.path.  This
    list is yet another new object in the sys.module: sys.meta_path.
    Currently, this list is empty by default, and frozen and builtin
    module imports are done after traversing sys.meta_path, but still
    before sys.path.  (Again, later we can add real frozen, builtin and
    sys.path importer objects on sys.meta_path, allowing for some extra
    flexibility, but this could be done as a "phase 2" project, possibly
    for Python 2.4.  It would be the finishing touch as then *every*
    import would go through sys.meta_path, making it the central import
    dispatcher.)

    As a bonus, the idea from the second paragraph of this section was
    implemented after all: a sys.path item may *be* an importer object.
    This use is discouraged for general purpose code, but it's very
    convenient, for experimentation as well as for projects of which
    it's known that no component wrongly assumes that sys.path items are
    strings.


Specification part 1: The Importer Protocol

    This PEP introduces a new protocol: the "Importer Protocol".  It is
    important to understand the context in which the protocol operates,
    so here is a brief overview of the outer shells of the import
    mechanism.

    When an import statement is encountered, the interpreter looks up
    the __import__ function in the builtin name space.  __import__ is
    then called with four arguments, amongst which are the name of the
    module being imported (may be a dotted name) and a reference to the
    current global namespace.

    The builtin __import__ function (known as PyImport_ImportModuleEx in
    import.c) will then check to see whether the module doing the import
    is a package by looking for a __path__ variable in the current
    global namespace.  If it is indeed a package, it first tries to do
    the import relative to the package.  For example if a package named
    "spam" does "import eggs", it will first look for a module named
    "spam.eggs".  If that fails, the import continues as an absolute
    import: it will look for a module named "eggs".  Dotted name imports
    work pretty much the same: if package "spam" does "import
    eggs.bacon", first "spam.eggs.bacon" is tried, and only if that
    fails "eggs.bacon" is tried.

    Deeper down in the mechanism, a dotted name import is split up by
    its components.  For "import spam.ham", first an "import spam" is
    done, and only when that succeeds is "ham" imported as a submodule
    of "spam".

    The Importer Protocol operates at this level of *individual*
    imports.  By the time an importer gets a request for "spam.ham",
    module "spam" has already been imported.

    The protocol involves two objects: an importer and a loader.  An
    importer object has a single method:

        importer.find_module(fullname)

    This method returns a loader object if the module was found, or None
    if it wasn't.  If find_module() raises an exception, it will be
    propagated to the caller, aborting the import.

    A loader object also has one method:

        loader.load_module(fullname)

    This method returns the loaded module.  In many cases the importer
    and loader can be one and the same object: importer.find_module()
    would just return self.

    The 'fullname' argument of both methods is the fully qualified
    module name, for example "spam.eggs.ham".  As explained above, when
    importer.find_module("spam.eggs.ham") is called, "spam.eggs" has
    already been imported and added to sys.modules.  However, the
    find_module() method isn't neccesarily always called during an
    actual import: meta tools that analyze import dependencies (such as
    freeze, Installer or py2exe) don't actually load modules, so an
    importer shouldn't *depend* on the parent package being available in
    sys.modules.

    The load_module() method has a few responsibilities that it must
    fulfill *before* it runs any code:

    - It must create the module object.  From Python this can be done
      via the new.module() function, the imp.new_module() function or
      via the module type object; from C with the PyModule_New()
      function or the PyImport_ModuleAdd() function.  The latter also
      does the following step:

    - It must add the module to sys.modules.  This is crucial because
      the module code may (directly or indirectly) import itself; adding
      it to sys.modules beforehand prevents unbounded recursion in the
      worst case and multiple loading in the best.

    - The __file__ attribute must be set.  This must be a string, but it
      may be a dummy value, for example "<frozen>".  The priviledge of
      not having a __file__ attribute at all is reserved for builtin
      modules.

    - If it's a package, the __path__ variable must be set.  This must
      be a list, but may be empty if __path__ has no further
      significance to the importer (more on this later).

    - It should add an __importer__ attribute to the module, set to the
      loader object.  This is mostly for introspection, but can be used
      for importer-specific extra's, for example getting data associated
      with an importer.

    If the module is a Python module (as opposed to a builtin module or
    an dynamically loaded extension), it should execute the module's
    code in the module's global name space (module.__dict__).

    Here is a minimal pattern for a load_module() method:

        def load_module(self, fullname):
            ispkg, code = self._get_code(fullname)
            mod = imp.new_module(fullname)
            sys.modules[fullname] = mod
            mod.__file__ = "<%s>" % self.__class__.__name__
            mod.__importer__ = self
            if ispkg:
                mod.__path__ = []
            exec code in mod.__dict__
            return mod


Specification part 2: Registering Hooks

    There are two types of import hooks: Meta hooks and Path hooks.
    Meta hooks are called at the start of import processing, before any
    other import processing (so that meta hooks can override sys.path
    processing, or frozen modules, or even builtin modules).  To
    register a meta hook, simply add the importer object to
    sys.meta_path (the list of registered meta hooks).

    Path hooks are called as part of sys.path (or package __path__)
    processing, at the point where their associated path item is
    encountered.  A path hook can be registered in either of two ways:

    - By simply including an importer object directly on the path.
      This approach is discouraged for general purpose hooks, as
      existing code may not be expecting non-strings to exist on
      sys.path.

    - By registering an importer factory in sys.path_hooks.

    sys.path_hooks is a list of callables, which will be checked in
    sequence to determine if they can handle a given path item.  The
    callable is called with one argument, the path item.  The callable
    must raise ImportError if it is unable to handle the path item, and
    return an importer object if it can handle the path item.  The
    callable is typically the class of the import hook, and hence the
    class __init__ method is called.  (This is also the reason why it
    should raise ImportError: an __init__ method can't return anything.)

    The results of path hook checks are cached in
    sys.path_importer_cache, which is a dictionary mapping path entries
    to importer objects.  The cache is checked before sys.path_hooks is
    scanned.  If it is necessary to force a rescan of sys.path_hooks, it
    is possible to manually clear all or part of
    sys.path_importer_cache.

    Just like sys.path itself, the new sys variables must have specific
    types:

        sys.meta_path and sys.path_hooks must be Python lists.
        sys.path_importer_cache must be a Python dict.

    Modifying these variables in place is allowed, as is replacing them
    with new objects.


Packages and the role of __path__

    If a module has a __path__ attribute, the import mechanism will
    treat it as a package.  The __path__ variable is used instead of
    sys.path when importing submodules of the package.  The rules for
    sys.path therefore also apply to pkg.__path__.  So sys.path_hooks is
    also consulted when pkg.__path__ is traversed and importer objects
    as path items are also allowed (yet, are discouraged for the same
    reasons as they are discouraged on sys.path, at least for general
    purpose code).  Meta importers don't neccesarily use sys.path at all
    to do their work and therefore may also ignore the value of
    pkg.__path__.  In this case it is still advised to set it to list,
    which can be empty.


Integration with the 'imp' module

    The new import hooks are not easily integrated in the existing
    imp.find_module() and imp.load_module() calls.  It's questionable
    whether it's possible at all without breaking code; it is better to
    simply add a new function to the imp module.  The meaning of the
    existing imp.find_module() and imp.load_module() calls changes from:
    "they expose the builtin import mechanism" to "they expose the basic
    *unhooked* builtin import mechanism".  They simply won't invoke any
    import hooks.  A new imp module function is proposed under the name
    "find_module2", with is used like the following pattern:

        loader = imp.find_module2(fullname, path)
        if loader is not None:
            loader.load_module(fullname)

    In the case of a "basic" import, one the imp.find_module() function
    would handle, the loader object would be a wrapper for the current
    output of imp.find_module(), and loader.load_module() would call
    imp.load_module() with that output.

    Note that this wrapper is currently not yet implemented, although a
    Python prototype exists in the test_importhooks.py script (the
    ImpWrapper class) included with the patch.


Open Issues

    The new hook method allows for the possibility of objects other than
    strings appearing on sys.path.  Existing code is entitled to assume
    that sys.path only contains strings (the Python documentation states
    this).  It is not clear if this will cause significant breakage.  In
    particular, it is much less clear that code is entitled to assume
    that sys.path contains a list of *directory names* - most code which
    assumes that sys.path items contain strings also rely on this extra
    assumption, and so could be considered as broken (or at least "not
    robust") already.

    Modules often need supporting data files to do their job,
    particularly in the case of complex packages or full applications.
    Current practice is generally to locate such files via sys.path (or
    a package __path__ variable).  This approach will not work, in
    general, for modules loaded via an import hook.

    There are a number of possible ways to address this problem:

    - "Don't do that".  If a package needs to locate data files via its
      __path__, it is not suitable for loading via an import hook.  The
      package can still be located on a directory in sys.path, as at
      present, so this should not be seen as a major issue.

    - Locate data files from a standard location, rather than relative
      to the module file.  A relatively simple approach (which is
      supported by distutils) would be to locate data files based on
      sys.prefix (or sys.exec_prefix).  For example, looking in
      os.path.join(sys.prefix, "data", package_name).

    - Import hooks could offer a standard way of getting at datafiles
      relative to the module file.  The standard zipimport object
      provides a method get_data(name) which returns the content of the
      "file" called name, as a string.  To allow modules to get at the
      importer object, zipimport also adds an attribute "__importer__"
      to the module, containing the zipimport object used to load the
      module.  If such an approach is used, it is important that client
      code takes care not to break if the get_data method (or the
      __importer__ attribute) is not available, so it is not clear that
      this approach offers a general answer to the problem.

    Requiring loaders to set the module's __importer__ attribute means
    that the loader will not get thrown away once the load is complete.
    This increases memory usage, and stops loaders from being
    lightweight, "throwaway" objects.  As loader objects are not
    required to offer any useful functionality (any such functionality,
    such as the zipimport get_data() method mentioned above, is
    optional) it is not clear that the __importer__ attribute will be
    helpful, in practice.

    On the other hand, importer objects are mostly permanent, as they
    live or are kept alive on sys.meta_path, sys.path_importer_cache or
    sys.path, so for a loader to keep a reference to the importer costs
    us nothing extra.  Whether loaders will ever need to carry so much
    independent state for this to become a real issue is questionable.


Implementation

    A C implementation is available as SourceForge patch 652586.
    http://www.python.org/sf/652586


References

    [1] Installer by Gordon McMillan
    http://www.mcmillan-inc.com/install1.html
    [2] PEP 273, Import Modules from Zip Archives, Ahlstrom
    http://www.python.org/peps/pep-0273.html
    [3] The Freeze tool
    Tools/freeze/ in a Python source distribution
    [4] Squeeze
    http://starship.python.net/crew/fredrik/ipa/squeeze.htm
    [5] py2exe by Thomas Heller
    http://py2exe.sourceforge.net/
    [6] imp.set_frozenmodules() patch
    http://www.python.org/sf/642578


Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:


From guido@python.org  Thu Dec 19 14:39:58 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 19 Dec 2002 09:39:58 -0500
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: Your message of "Thu, 19 Dec 2002 08:16:29 +0100."
 <005201c2a72e$8ec4c6b0$ced241d5@hagrid>
References: <r01050400-1022-E00BBF9F12D811D7A7C4003065D5E7E4@[10.0.0.1]>
 <005201c2a72e$8ec4c6b0$ced241d5@hagrid>
Message-ID: <200212191439.gBJEdwo23277@odiug.zope.com>

> some of us has been shipping deployment tools for many years, and
> have a pretty good understanding of what optional extensions we and
> our customers have needed over the years...

Would you mind summarizing what you think is needed?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Thu Dec 19 14:47:37 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 19 Dec 2002 09:47:37 -0500
Subject: [Python-Dev] Zipping Zope3
In-Reply-To: Your message of "Thu, 19 Dec 2002 13:56:49 +0100."
 <564D985C-1351-11D7-84BC-003065517236@oratrix.com>
References: <564D985C-1351-11D7-84BC-003065517236@oratrix.com>
Message-ID: <200212191447.gBJElbc23342@odiug.zope.com>

> > Also: *anything* that makes zipimporter closer to a file system is
> > going at it from the wrong direction. We *first* need a virtual
> > file system, then an import hook that can import from it, and
> > *then* a zip back end...

Note that there's a design for a VFS API in ihooks.py.  Worth checking
out.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Thu Dec 19 14:52:39 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 19 Dec 2002 09:52:39 -0500
Subject: [Python-Dev] Mail to fredrik@pythonware.com bounces
Message-ID: <200212191452.gBJEqdn23797@odiug.zope.com>

Has the effbot decided he's not interested in feedback any more? :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)

------- Forwarded Message

Date:    Thu, 19 Dec 2002 09:41:46 -0500
From:    Mail Delivery Subsystem <MAILER-DAEMON@odiug.zope.com>
To:      <guido@odiug.zope.com>
Subject: Returned mail: see transcript for details

This is a MIME-encapsulated message

- --gBJEfko23279.1040308906/odiug.zope.com

The original message was received at Thu, 19 Dec 2002 09:39:59 -0500
from guido@localhost

   ----- The following addresses had permanent fatal errors -----
<fredrik@pythonware.com>
    (reason: 550 <fredrik@pythonware.com>... denied)

   ----- Transcript of session follows -----
... while talking to mailgw.algonet.se.:
>>> RCPT To:<fredrik@pythonware.com>
<<< 550 <fredrik@pythonware.com>... denied
550 5.1.1 <fredrik@pythonware.com>... User unknown

- --gBJEfko23279.1040308906/odiug.zope.com
Content-Type: message/delivery-status

Reporting-MTA: dns; odiug.zope.com
Arrival-Date: Thu, 19 Dec 2002 09:39:59 -0500

Final-Recipient: RFC822; fredrik@pythonware.com
Action: failed
Status: 5.1.1
Remote-MTA: DNS; mailgw.algonet.se
Diagnostic-Code: SMTP; 550 <fredrik@pythonware.com>... denied
Last-Attempt-Date: Thu, 19 Dec 2002 09:41:03 -0500

- --gBJEfko23279.1040308906/odiug.zope.com
Content-Type: message/rfc822

Return-Path: <guido@odiug.zope.com>
Received: from guido (guido@localhost)
	by odiug.zope.com (8.11.0/8.11.0) with ESMTP id gBJEdwo23277;
	Thu, 19 Dec 2002 09:39:59 -0500
Message-Id: <200212191439.gBJEdwo23277@odiug.zope.com>
To: "Fredrik Lundh" <fredrik@pythonware.com>
cc: "Jack Jansen" <Jack.Jansen@oratrix.com>,
   "Samuele Pedroni" <pedronis@bluewin.ch>,
   "Moore,
    Paul" <Paul.Moore@atosorigin.com>, python-dev@python.org
Subject: Re: [Python-Dev] Zipping Zope3
In-reply-to: Your message of "Thu, 19 Dec 2002 08:16:29 +0100."
             <005201c2a72e$8ec4c6b0$ced241d5@hagrid> 
References: <r01050400-1022-E00BBF9F12D811D7A7C4003065D5E7E4@[10.0.0.1]>  
            <005201c2a72e$8ec4c6b0$ced241d5@hagrid> 
From: Guido van Rossum <guido@python.org>
Date: Thu, 19 Dec 2002 09:39:58 -0500
Sender: guido@odiug.zope.com

> some of us has been shipping deployment tools for many years, and
> have a pretty good understanding of what optional extensions we and
> our customers have needed over the years...

Would you mind summarizing what you think is needed?

- --Guido van Rossum (home page: http://www.python.org/~guido/)

- --gBJEfko23279.1040308906/odiug.zope.com--

------- End of Forwarded Message



From neal@metaslash.com  Thu Dec 19 14:53:46 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Thu, 19 Dec 2002 09:53:46 -0500
Subject: [Python-Dev] deprecating APIs
In-Reply-To: <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net>
References: <20021217014935.GQ16807@epoch.metaslash.com> <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20021219145346.GG12063@epoch.metaslash.com>

On Wed, Dec 18, 2002 at 10:54:34PM -0500, Guido van Rossum wrote:
> 
> What does this do under GCC?  I suppose it issues a warning when you
> use the deprecated API?

I modified abstract.h:

-     PyAPI_FUNC(int) PyNumber_Check(PyObject *o);
+     PyAPI_FUNC(int) PyNumber_Check(PyObject *o) Py_DEPRECATED(2.3);

I used PyNumber_Check in selectmodule.c and got:

Modules/selectmodule.c: In function `select_select':
Modules/selectmodule.c:211: warning: `PyNumber_Check' is deprecated 
                                (declared at Include/abstract.h:534)

Neal


From guido@python.org  Thu Dec 19 14:59:10 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 19 Dec 2002 09:59:10 -0500
Subject: [Python-Dev] deprecating APIs
In-Reply-To: Your message of "Thu, 19 Dec 2002 09:53:46 EST."
 <20021219145346.GG12063@epoch.metaslash.com>
References: <20021217014935.GQ16807@epoch.metaslash.com> <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net>
 <20021219145346.GG12063@epoch.metaslash.com>
Message-ID: <200212191459.gBJExAa25897@odiug.zope.com>

> > What does this do under GCC?  I suppose it issues a warning when you
> > use the deprecated API?
> 
> I modified abstract.h:
> 
> -     PyAPI_FUNC(int) PyNumber_Check(PyObject *o);
> +     PyAPI_FUNC(int) PyNumber_Check(PyObject *o) Py_DEPRECATED(2.3);
> 
> I used PyNumber_Check in selectmodule.c and got:
> 
> Modules/selectmodule.c: In function `select_select':
> Modules/selectmodule.c:211: warning: `PyNumber_Check' is deprecated 
>                                 (declared at Include/abstract.h:534)

Neat!  +1.

Now it's time to make a list of APIs we want to declare deprecated.
(Some have probably been deprecated for quite a while...)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From neal@metaslash.com  Thu Dec 19 15:46:41 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Thu, 19 Dec 2002 10:46:41 -0500
Subject: [Python-Dev] deprecating APIs
In-Reply-To: <200212191459.gBJExAa25897@odiug.zope.com>
References: <20021217014935.GQ16807@epoch.metaslash.com> <200212190354.gBJ3sYx06332@pcp02138704pcs.reston01.va.comcast.net> <20021219145346.GG12063@epoch.metaslash.com> <200212191459.gBJExAa25897@odiug.zope.com>
Message-ID: <20021219154641.GH12063@epoch.metaslash.com>

On Thu, Dec 19, 2002 at 09:59:10AM -0500, Guido van Rossum wrote:
> 
> Now it's time to make a list of APIs we want to declare deprecated.
> (Some have probably been deprecated for quite a while...)

There are only 847. :-)

I don't think this really helps... But for usage counts (by grep):

        http://www.metaslash.com/py/api-counts.txt

If nothing else, it's a quick list of names to look over.

Neal


From just@letterror.com  Thu Dec 19 18:14:55 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 19 Dec 2002 19:14:55 +0100
Subject: [Python-Dev] MacOSX build broken
Message-ID: <r01050400-1022-CA8AC62F137D11D7A7C4003065D5E7E4@[10.0.0.1]>

A plain ./configure ends like this, without creating a Makefile:

checking for --with-universal-newlines... --without-universal-newlines
is unsupported, see README

The README confirms --with-universal-newlines is no longer supported.
Yet I didn't specify --with-universal-newlines and the README doesn't
tell me what to do either.

What's up?

Just


From martin@v.loewis.de  Thu Dec 19 19:06:36 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 19 Dec 2002 20:06:36 +0100
Subject: [Python-Dev] MacOSX build broken
In-Reply-To: <r01050400-1022-CA8AC62F137D11D7A7C4003065D5E7E4@[10.0.0.1]>
References: <r01050400-1022-CA8AC62F137D11D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <m33cothlqb.fsf@mira.informatik.hu-berlin.de>

Just van Rossum <just@letterror.com> writes:

> What's up?

It was a bug; it should be fixed now.

Regards,
Martin


From brett@python.org  Thu Dec 19 21:10:11 2002
From: brett@python.org (Brett Cannon)
Date: Thu, 19 Dec 2002 13:10:11 -0800 (PST)
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
In-Reply-To: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>
References: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <Pine.SOL.4.51.0212191305180.6740@death.OCF.Berkeley.EDU>

Now why couldn't you guys have finished this *before* the summary went
out?!?  Then I could have just  said, "python-dev talked a lot about a new
import system.  Read the PEP if you care".  =)

Good PEP, all kiddding aside.  Both of you did a good job covering the
bases in an easy way to comprehend.

-Brett

P.S.: Removed cc: to peps@python.org


From just@letterror.com  Thu Dec 19 21:34:56 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 19 Dec 2002 22:34:56 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <Pine.SOL.4.51.0212191305180.6740@death.OCF.Berkeley.EDU>
Message-ID: <r01050400-1022-3E19BD88139A11D7A7C4003065D5E7E4@[10.0.0.1]>

Brett Cannon wrote:

> Now why couldn't you guys have finished this *before* the summary
> went out?!?  Then I could have just  said, "python-dev talked a lot
> about a new import system.  Read the PEP if you care".  =)

;-) Now if only we got a PEP number...

> Good PEP, all kiddding aside.  Both of you did a good job covering
> the bases in an easy way to comprehend.

Thanks.

Just


From Jack.Jansen@oratrix.com  Thu Dec 19 23:03:15 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Fri, 20 Dec 2002 00:03:15 +0100
Subject: [Python-Dev] Strange crash in import
In-Reply-To: <911129A1-1208-11D7-BC1B-000A27B19B96@oratrix.com>
Message-ID: <0DF13AB8-13A6-11D7-88F3-000A27B19B96@oratrix.com>

On dinsdag, dec 17, 2002, at 22:43 Europe/Amsterdam, Jack Jansen wrote:

> I'm getting a strange crash in import, and it turns out that there's a 
> refcount 0 object on sys.path.

Just in case anyone is interested: this turned out to be my mistake, I 
overly enthusiastically cleared out memory beyond my own object.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From aahz@pythoncraft.com  Thu Dec 19 23:24:54 2002
From: aahz@pythoncraft.com (Aahz)
Date: Thu, 19 Dec 2002 18:24:54 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>
References: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <20021219232454.GA24288@panix.com>

First of all, excellent job!  I've been feeling somewhat frustrated by
the number of directions this discussion has taken, and I'm glad to have
something that's clear and understandable.

One typo nitpick, one real question:

On Thu, Dec 19, 2002, Just van Rossum wrote:
>
>     - The __file__ attribute must be set.  This must be a string, but it
>       may be a dummy value, for example "<frozen>".  The priviledge of
>       not having a __file__ attribute at all is reserved for builtin
>       modules.

"privilege"

>     sys.path_hooks is a list of callables, which will be checked in
>     sequence to determine if they can handle a given path item.  The
>     callable is called with one argument, the path item.  The callable
>     must raise ImportError if it is unable to handle the path item, and
>     return an importer object if it can handle the path item.  The
>     callable is typically the class of the import hook, and hence the
>     class __init__ method is called.  (This is also the reason why it
>     should raise ImportError: an __init__ method can't return anything.)

Any reason we can't require importer objects to be new-style classes,
which could then use __new__?  (I vaguely recall some discussion about
this, but it should be documented in the PEP.)
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From airkjh@i-on.net  Fri Dec 20 06:52:31 2002
From: airkjh@i-on.net (=?ks_c_5601-1987?B?sejA58f1?=)
Date: Fri, 20 Dec 2002 15:52:31 +0900
Subject: [Python-Dev] Hi everyone!!
Message-ID: <CCB04B39E4234E449A1304667D40F37A075552@61-250-201-141.rev.krline.net>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C2A7F4.5E05D1F6
Content-Type: text/plain;
	charset="ks_c_5601-1987"
Content-Transfer-Encoding: quoted-printable

I=A1=AFm a new member of python mailing list.
I=A1=AFve developed some programs with python for 2 years ( especially, =
Database programs using Oracle , MS-SQL , DB2 )
=20
I hope to be a good helper and friend to you
Have a nice day
=20
------------------------------------
Kim Jae Hyuk
Customer Engineer
Development Division
I-ON Communications
TEL  : 82-2-574-1080
Mail : airkjh@i-on.net
------------------------------------
=20

------_=_NextPart_001_01C2A7F4.5E05D1F6
Content-Type: text/html;
	charset="ks_c_5601-1987"
Content-Transfer-Encoding: quoted-printable

<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dks_c_5601-1987">


<meta name=3DProgId content=3DWord.Document>
<meta name=3DGenerator content=3D"Microsoft Word 10">
<meta name=3DOriginator content=3D"Microsoft Word 10">
<link rel=3DFile-List href=3D"cid:filelist.xml@01C2A83F.CD91DDD0">
<!--[if gte mso 9]><xml>
 <o:OfficeDocumentSettings>
  <o:DoNotRelyOnCSS/>
 </o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
 <w:WordDocument>
  <w:SpellingState>Clean</w:SpellingState>
  <w:GrammarState>Clean</w:GrammarState>
  <w:DocumentKind>DocumentEmail</w:DocumentKind>
  <w:EnvelopeVis/>
  =
<w:DisplayHorizontalDrawingGridEvery>0</w:DisplayHorizontalDrawingGridEve=
ry>
  =
<w:DisplayVerticalDrawingGridEvery>2</w:DisplayVerticalDrawingGridEvery>
  <w:Compatibility>
   <w:SpaceForUL/>
   <w:BalanceSingleByteDoubleByteWidth/>
   <w:DoNotLeaveBackslashAlone/>
   <w:ULTrailSpace/>
   <w:DoNotExpandShiftReturn/>
   <w:AdjustLineHeightInTable/>
   <w:BreakWrappedTables/>
   <w:SnapToGridInCell/>
   <w:WrapTextWithPunct/>
   <w:UseAsianBreakRules/>
   <w:UseFELayout/>
  </w:Compatibility>
  <w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
 </w:WordDocument>
</xml><![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:=B9=D9=C5=C1;
	panose-1:2 3 6 0 0 1 1 1 1 1;
	mso-font-alt:Batang;
	mso-font-charset:129;
	mso-generic-font-family:roman;
	mso-font-pitch:variable;
	mso-font-signature:-1342176593 1775729915 48 0 524447 0;}
@font-face
	{font-family:=B1=BC=B8=B2;
	panose-1:2 11 6 0 0 1 1 1 1 1;
	mso-font-alt:Gulim;
	mso-font-charset:129;
	mso-generic-font-family:modern;
	mso-font-pitch:variable;
	mso-font-signature:-1342176593 1775729915 48 0 524447 0;}
@font-face
	{font-family:"\@=B9=D9=C5=C1";
	panose-1:2 3 6 0 0 1 1 1 1 1;
	mso-font-charset:129;
	mso-generic-font-family:roman;
	mso-font-pitch:variable;
	mso-font-signature:-1342176593 1775729915 48 0 524447 0;}
@font-face
	{font-family:=B5=B8=BF=F2=C3=BC;
	panose-1:2 11 6 9 0 1 1 1 1 1;
	mso-font-charset:129;
	mso-generic-font-family:modern;
	mso-font-pitch:fixed;
	mso-font-signature:-1342176593 1775729915 48 0 524447 0;}
@font-face
	{font-family:"\@=B5=B8=BF=F2=C3=BC";
	panose-1:2 11 6 9 0 1 1 1 1 1;
	mso-font-charset:129;
	mso-generic-font-family:modern;
	mso-font-pitch:fixed;
	mso-font-signature:-1342176593 1775729915 48 0 524447 0;}
@font-face
	{font-family:"\@=B1=BC=B8=B2";
	panose-1:2 11 6 0 0 1 1 1 1 1;
	mso-font-charset:129;
	mso-generic-font-family:modern;
	mso-font-pitch:variable;
	mso-font-signature:-1342176593 1775729915 48 0 524447 0;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{mso-style-parent:"";
	margin:0cm;
	margin-bottom:.0001pt;
	text-align:justify;
	text-justify:inter-ideograph;
	mso-pagination:none;
	text-autospace:none;
	word-break:break-hangul;
	font-size:10.0pt;
	mso-bidi-font-size:12.0pt;
	font-family:=B9=D9=C5=C1;
	mso-hansi-font-family:"Times New Roman";
	mso-bidi-font-family:"Times New Roman";
	mso-font-kerning:1.0pt;}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;
	text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;
	text-underline:single;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	mso-style-noshow:yes;
	mso-ansi-font-size:10.0pt;
	mso-bidi-font-size:10.0pt;
	font-family:=B1=BC=B8=B2;
	mso-ascii-font-family:=B1=BC=B8=B2;
	mso-fareast-font-family:=B1=BC=B8=B2;
	mso-hansi-font-family:Arial;
	mso-bidi-font-family:Arial;
	color:windowtext;}
span.GramE
	{mso-style-name:"";
	mso-gram-e:yes;}
 /* Page Definitions */
 @page
	{mso-page-border-surround-header:no;
	mso-page-border-surround-footer:no;}
@page Section1
	{size:595.3pt 841.9pt;
	margin:99.25pt 3.0cm 3.0cm 3.0cm;
	mso-header-margin:42.55pt;
	mso-footer-margin:49.6pt;
	mso-paper-source:0;
	layout-grid:18.0pt;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 10]>
<style>
 /* Style Definitions */=20
 table.MsoNormalTable
	{mso-style-name:"=C7=A5=C1=D8 =C7=A5";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-parent:"";
	mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
	mso-para-margin:0cm;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:10.0pt;
	font-family:"Times New Roman";}
</style>
<![endif]-->
</head>

<body lang=3DKO link=3Dblue vlink=3Dpurple =
style=3D'tab-interval:40.0pt'>

<div class=3DSection1 style=3D'layout-grid:18.0pt'>

<p class=3DMsoNormal><font size=3D2 face=3D=B1=BC=B8=B2><span =
lang=3DEN-US style=3D'font-size:10.0pt;
mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-font-family:=
Arial;
mso-bidi-font-family:Arial'>I</span></font><font face=3DArial><span =
lang=3DEN-US
style=3D'mso-bidi-font-size:10.0pt;font-family:Arial;mso-ascii-font-famil=
y:=B1=BC=B8=B2;
mso-fareast-font-family:=B1=BC=B8=B2'>=A1=AF</span></font><font =
face=3D=B1=BC=B8=B2><span lang=3DEN-US
style=3D'mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-fon=
t-family:Arial;
mso-bidi-font-family:Arial'>m a new member of python mailing =
list.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B1=BC=B8=B2><span =
lang=3DEN-US style=3D'font-size:10.0pt;
mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-font-family:=
Arial;
mso-bidi-font-family:Arial'>I</span></font><font face=3DArial><span =
lang=3DEN-US
style=3D'mso-bidi-font-size:10.0pt;font-family:Arial;mso-ascii-font-famil=
y:=B1=BC=B8=B2;
mso-fareast-font-family:=B1=BC=B8=B2'>=A1=AF</span></font><font =
face=3D=B1=BC=B8=B2><span lang=3DEN-US
style=3D'mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-fon=
t-family:Arial;
mso-bidi-font-family:Arial'>ve developed some programs with python for 2 =
years <span
class=3DGramE>( especially</span>, Database programs using Oracle , =
MS-SQL , DB2
)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B1=BC=B8=B2><span =
lang=3DEN-US style=3D'font-size:10.0pt;
mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-font-family:=
Arial;
mso-bidi-font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B1=BC=B8=B2><span =
lang=3DEN-US style=3D'font-size:10.0pt;
mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-font-family:=
Arial;
mso-bidi-font-family:Arial'>I hope to be a good helper and friend to =
you<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B1=BC=B8=B2><span =
lang=3DEN-US style=3D'font-size:10.0pt;
mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-font-family:=
Arial;
mso-bidi-font-family:Arial'>Have a nice day<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B1=BC=B8=B2><span =
lang=3DEN-US style=3D'font-size:10.0pt;
mso-bidi-font-size:10.0pt;font-family:=B1=BC=B8=B2;mso-hansi-font-family:=
Arial;
mso-bidi-font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>------------------------------------<=
o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>Kim
Jae Hyuk<o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>Customer
Engineer<o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>Development
Division<o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>I-ON
Communications<o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>TEL<span
style=3D'mso-spacerun:yes'>&nbsp; </span>: =
82-2-574-1080<o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>Mail
: airkjh@i-on.net<o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B5=B8=BF=F2=C3=BC><span =
lang=3DEN-US style=3D'font-size:
10.0pt;font-family:=B5=B8=BF=F2=C3=BC;mso-no-proof:yes'><span =
style=3D'mso-bidi-font-size:12.0pt'>------------------------------------<=
o:p></o:p></span></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D=B9=D9=C5=C1><span =
lang=3DEN-US><o:p>&nbsp;</o:p></span></font></p>

</div>

</body>

</html>

------_=_NextPart_001_01C2A7F4.5E05D1F6--


From fredrik@pythonware.com  Fri Dec 20 07:47:26 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Fri, 20 Dec 2002 08:47:26 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
References: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]> <20021219232454.GA24288@panix.com>
Message-ID: <004201c2a7fc$0c30ff10$ced241d5@hagrid>

Aahz keeps confusing me:

> Any reason we can't require importer objects to be new-style classes,
> which could then use __new__?  (I vaguely recall some discussion about
> this, but it should be documented in the PEP.)

any reason we *should* require importer objects to be new-style
classes?

if you think raising exceptions is a problem, please say so.

if you have a valid argument for using return values instead of exceptions
(it's not like people will be installing thousands of hooks, so it's not obvious
what the problem is), is there any reason not to allow people to use *any*
callable object that can return a value?  (including three-line wrappers for
pre-2.2 classes).

</F>



From Paul.Moore@atosorigin.com  Fri Dec 20 09:39:07 2002
From: Paul.Moore@atosorigin.com (Moore, Paul)
Date: Fri, 20 Dec 2002 09:39:07 -0000
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D800@UKDCX001.uk.int.atosorigin.com>

Aahz said:
> Any reason we can't require importer objects to be new-style
> classes, which could then use __new__?  (I vaguely recall
> some discussion about this, but it should be documented in
> the PEP.)

Basically, why bother? It just makes writing importers (mildly)
more complex (most people know about __init__ whereas __new__
is ... err ... newer), where the existing proposal works fine.

But you're right, a comment in the PEP would be good.

Paul


From aahz@pythoncraft.com  Fri Dec 20 14:02:34 2002
From: aahz@pythoncraft.com (Aahz)
Date: Fri, 20 Dec 2002 09:02:34 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <004201c2a7fc$0c30ff10$ced241d5@hagrid>
References: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]> <20021219232454.GA24288@panix.com> <004201c2a7fc$0c30ff10$ced241d5@hagrid>
Message-ID: <20021220140234.GA13980@panix.com>

On Fri, Dec 20, 2002, Fredrik Lundh wrote:
>
> if you have a valid argument for using return values instead of
> exceptions (it's not like people will be installing thousands of
> hooks, so it's not obvious what the problem is), is there any reason
> not to allow people to use *any* callable object that can return a
> value? (including three-line wrappers for pre-2.2 classes).

Good point.  That should go in the PEP.  

(In case it wasn't clear, I was arguing neither for nor against
new-style classes, I just wanted it documented in the PEP since it was
discussed.)

Hmmmm....  I seem to also recall seeing discussion about the callable
raising something other than ImportError when it just can't handle a
path element.  What happened to that idea?  That would allow the hook
handler to distinguish between "I had an error in handling this path"
and "I can't handle this path".
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"To me vi is Zen.  To use vi is to practice zen.  Every command is a
koan.  Profound to the user, unintelligible to the uninitiated.  You
discover truth everytime you use it."  --reddy@lion.austin.ibm.com


From just@letterror.com  Fri Dec 20 14:08:58 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 20 Dec 2002 15:08:58 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <20021220140234.GA13980@panix.com>
Message-ID: <r01050400-1022-9AD7DC70142411D7A7C4003065D5E7E4@[10.0.0.1]>

Aahz wrote:

> Hmmmm....  I seem to also recall seeing discussion about the callable
> raising something other than ImportError when it just can't handle a
> path element.  What happened to that idea?  That would allow the hook
> handler to distinguish between "I had an error in handling this path"
> and "I can't handle this path".

I argued that it is unlikely that calling the hook would cause an
import, so raising ImportError should be just fine. I don't recall
getting any opposition for that. It's now specced as "must raise
ImportError". I really think the issue is too minor to warrant a new
exception.

Just


From jim@interet.com  Fri Dec 20 17:08:07 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 20 Dec 2002 12:08:07 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
References: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <3E034E77.8090609@interet.com>

Just van Rossum wrote:
> PEP: XXX
> Title: New Import Hooks

First, I commend Just and Paul for their work.  It is not easy
sloughing through Python imports.  And the background on how
imports work and the desired changes are valuable.  I am the
author of the competing PEP 273 implementation, and if Just's
implementation can provide a better implementation, I won't
mind.

But the changes to the imp module IMHO are ill advised.  I am
posting an alternative here, and will post other objections
separately.

> Integration with the 'imp' module
> 
>     The new import hooks are not easily integrated in the existing
>     imp.find_module() and imp.load_module() calls.  It's questionable
>     whether it's possible at all without breaking code; it is better to
>     simply add a new function to the imp module.

Although I originally found the find/load dichotomy annoying,
I now think it should be kept.  It solves the real
world problem of finding files which are not Python modules.
Currently Zope and others find configuration and data files
either by looping over sys.path themselves, or by looking
at the base name of module.__file__ (the file name of an
imported module).  Both fail for zip imports.

I too find the imp find/load function signatures hard to love.  But
I also don't want to break any code which depends on them, nor
make gratuitous changes which make them useless for zip imports.
I think the imp module should Just Work for zip imports.

I suggest we keep imp.find_module but add another argument:

      find_module(name [, path [, suffixes]])

where suffixes defaults to imp.get_suffixes().  This enables it
to be used to search for files with different suffixes, for
example suffixes=[".conf"].

To make this work, the returned file object would have to be
either a real file object, or another object with a read() method.
Python has lots of precedents for accepting file-like objects.  For
example, sys.stdout can be replaced with any object which has a
write method.

The returned file-like object must have a read() method and
a close() method.  It could also have a stat() method if it
is a zip file, because zip files record the file date, time
and size.

So you call file.read() for a configuration file, or pass
it to imp.load_module() if it is a Python module.

JimA



From jim@interet.com  Fri Dec 20 17:23:37 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 20 Dec 2002 12:23:37 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
References: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]> <3E034E77.8090609@interet.com>
Message-ID: <3E035219.6050007@interet.com>

James C. Ahlstrom wrote:

> to be used to search for files with different suffixes, for
> example suffixes=[".conf"].

Ooops, that should be
     suffixes = [(".conf", "r", 0)]

JimA



From just@letterror.com  Fri Dec 20 18:25:27 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 20 Dec 2002 19:25:27 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <3E034E77.8090609@interet.com>
Message-ID: <r01050400-1022-6B419BC4144811D7A7C4003065D5E7E4@[10.0.0.1]>

James C. Ahlstrom wrote:

> Although I originally found the find/load dichotomy annoying,
> I now think it should be kept.

It *is* kept, in every detail of the PEP.

> It solves the real
> world problem of finding files which are not Python modules.
> Currently Zope and others find configuration and data files
> either by looping over sys.path themselves, or by looking
> at the base name of module.__file__ (the file name of an
> imported module).  Both fail for zip imports.

That's why I introduced __importer__ and __importer__.get_data(). (The
details of their specification might change, though. Oh wait, get_data()
has no specification at all yet ;-)

> I too find the imp find/load function signatures hard to love.  But
> I also don't want to break any code which depends on them, nor
> make gratuitous changes which make them useless for zip imports.
> I think the imp module should Just Work for zip imports.

I didn't change anything there and I sure don't break any code, because
there's hardly any code out there that expects imp.find_module() to work
for zip files <0.5 wink>. However, for imp.find_module() to "just work"
and be *useful* for imports from importer objects, it will need to
return a loader object. I don't see how this can be done without
breaking imp.find_module(). I thought of returning a loader object
instead of a file object for "hooked" imports, but unless we add a dummy
close() method to all loader objects, this will break a common idiom:

    file, filename, stuff = imp.find_module(...)
    if file:
        file.close()

And that's just the beginning of the trouble. It's also rather hackish.
So I propose to add a new function to the imp module, as described in
the PEP. And no, this one doesn't solve the data file problem.

I could change imp.find_module() to return *something* for hooked
imports, but it won't be useful to do imports without a loader object.

> I suggest we keep imp.find_module but add another argument:
> 
>       find_module(name [, path [, suffixes]])
> 
> where suffixes defaults to imp.get_suffixes().  This enables it
> to be used to search for files with different suffixes, for
> example suffixes=[".conf"].

I'm not sure if I like this: imp.find_module() is designed for finding
modules, not arbitrary files. It's an interesting idea though, but this
would neccesarily complicate the importer protocol, as each importer
would have to deal with arbitrary suffixes. It implies file-system
semantics that importer objects can't neccesarily satisfy, eg. an
importer doesn't neccesarily deal with suffixes at *all*.

> To make this work, the returned file object would have to be
> either a real file object, or another object with a read() method.
> Python has lots of precedents for accepting file-like objects.  For
> example, sys.stdout can be replaced with any object which has a
> write method.
> 
> The returned file-like object must have a read() method and
> a close() method.  It could also have a stat() method if it
> is a zip file, because zip files record the file date, time
> and size.

(File objects don't have a stat method.)

> So you call file.read() for a configuration file, or pass
> it to imp.load_module() if it is a Python module.

I'll think about this some more, but I'm not yet convinced.

Just


From theller@python.net  Fri Dec 20 19:05:09 2002
From: theller@python.net (Thomas Heller)
Date: 20 Dec 2002 20:05:09 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <r01050400-1022-6B419BC4144811D7A7C4003065D5E7E4@[10.0.0.1]>
References: <r01050400-1022-6B419BC4144811D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <znr0zf2y.fsf@python.net>

> > I suggest we keep imp.find_module but add another argument:
> > 
> >       find_module(name [, path [, suffixes]])
> > 
> > where suffixes defaults to imp.get_suffixes().  This enables it
> > to be used to search for files with different suffixes, for
> > example suffixes=[".conf"].
> 
> I'm not sure if I like this: imp.find_module() is designed for finding
> modules, not arbitrary files. It's an interesting idea though, but this
> would neccesarily complicate the importer protocol, as each importer
> would have to deal with arbitrary suffixes. It implies file-system
> semantics that importer objects can't neccesarily satisfy, eg. an
> importer doesn't neccesarily deal with suffixes at *all*.

I also found this an interesting idea, first.

But then, how would this (working) code have to be written with the
new imp.find_module() function:

    def get_exe_bytes (self):
        # wininst.exe is in the same directory as this file
        directory = os.path.dirname(__file__)
        filename = os.path.join(directory, "wininst.exe")
        return open(filename, "rb").read()

Thomas



From just@letterror.com  Fri Dec 20 19:06:38 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 20 Dec 2002 20:06:38 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <3E034E77.8090609@interet.com>
Message-ID: <r01050400-1022-2E4EDCBE144E11D7A7C4003065D5E7E4@[10.0.0.1]>

James C. Ahlstrom wrote:

> I suggest we keep imp.find_module but add another argument:
> 
>       find_module(name [, path [, suffixes]])
> 
> where suffixes defaults to imp.get_suffixes().  This enables it
> to be used to search for files with different suffixes, for
> example suffixes=[".conf"].

One more thought: finding modules and finding data files is not the same
thing. Modules can be anywhere and must be accessible to anyone, but
data files belong to specific modules. Searching the entire sys.path for
a data file seems way too broad; there's got to be some information
available as to where the data is. Eg. I doubt it's a *feature* that
pdb.doc may also be found in site-packages.

Just

PS: Sorry for Cc-ing peps@python.org before, it was a bad idea to
crosspost the draft to python-dev and peps to begin with :-(


From just@letterror.com  Fri Dec 20 19:14:01 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 20 Dec 2002 20:14:01 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <znr0zf2y.fsf@python.net>
Message-ID: <r01050400-1022-34C0EBF2144F11D7A7C4003065D5E7E4@[10.0.0.1]>

Thomas Heller wrote:

> But then, how would this (working) code have to be written with the
> new imp.find_module() function:
> 
>     def get_exe_bytes (self):
>         # wininst.exe is in the same directory as this file
>         directory = os.path.dirname(__file__)
>         filename = os.path.join(directory, "wininst.exe")
>         return open(filename, "rb").read()

With the importer protocol (as I imagine it now, this doesn't match
zipimporter's current behavior yet) it could be written like this:

    def get_exe_bytes (self):
        if os.path.isfile(__file__):
            # wininst.exe is in the same directory as this file
            directory = os.path.dirname(__file__)
            filename = os.path.join(directory, "wininst.exe")
            return open(filename, "rb").read()
        else:
            path = __name__.replace(".", os.sep) + os.sep + \
                   "wininst.exe"
            return __importer__.get_data(path)

Just

PS: please take care to not Cc to peps@python.org any more in this
thread, I think I've done enough harm ;-)


From theller@python.net  Fri Dec 20 19:46:05 2002
From: theller@python.net (Thomas Heller)
Date: 20 Dec 2002 20:46:05 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <r01050400-1022-34C0EBF2144F11D7A7C4003065D5E7E4@[10.0.0.1]>
References: <r01050400-1022-34C0EBF2144F11D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <u1h8zd6q.fsf@python.net>

Just van Rossum <just@letterror.com> writes:

> Thomas Heller wrote:
> 
> > But then, how would this (working) code have to be written with the
> > new imp.find_module() function:
> > 
> >     def get_exe_bytes (self):
> >         # wininst.exe is in the same directory as this file
> >         directory = os.path.dirname(__file__)
> >         filename = os.path.join(directory, "wininst.exe")
> >         return open(filename, "rb").read()
> 
> With the importer protocol (as I imagine it now, this doesn't match
> zipimporter's current behavior yet) it could be written like this:
> 
>     def get_exe_bytes (self):
>         if os.path.isfile(__file__):
>             # wininst.exe is in the same directory as this file
>             directory = os.path.dirname(__file__)
>             filename = os.path.join(directory, "wininst.exe")
>             return open(filename, "rb").read()
>         else:
>             path = __name__.replace(".", os.sep) + os.sep + \
>                    "wininst.exe"
>             return __importer__.get_data(path)
> 
Looks ok to me, although I would prefer to write

        path = os.sep.join(__name__.split(".") + ["wininst.exe"])
or
        path = os.path.join(*__name__.split(".") + ["wininst.exe"])

I just wanted to make sure that I don't have to construct
the arguments for imp.find_module() from __file__.
> 
> PS: please take care to not Cc to peps@python.org any more in this
> thread, I think I've done enough harm ;-)

Sorry for that, I noticed it as it was already too late.

Thomas



From jim@interet.com  Fri Dec 20 20:22:22 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 20 Dec 2002 15:22:22 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
References: <r01050400-1022-6B419BC4144811D7A7C4003065D5E7E4@[10.0.0.1]> <znr0zf2y.fsf@python.net>
Message-ID: <3E037BFE.7020509@interet.com>

Thomas Heller wrote:

> But then, how would this (working) code have to be written with the
> new imp.find_module() function:
> 
>     def get_exe_bytes (self):
>         # wininst.exe is in the same directory as this file
>         directory = os.path.dirname(__file__)
>         filename = os.path.join(directory, "wininst.exe")
>         return open(filename, "rb").read()


It would look like this:

   suffix = [(".exe", "rb", 0)]
   file, path, descr = imp.find_module("wininst", __path__, suffix)
   data = file.read()
   file.close()
   return data

Note that for a zip file, __file__ is something like
      C:/Python/lib/zarchive.zip/subdir/mymod.pyc
and nothing is going to make code using __file__ Just Work.

Well, maybe you could strip the last component "mymod.pyc" and
use it as __path__ in a call to imp.find_module().  Not too
hard, and it works with files and zip archives.  But maybe not
with future ftp:// imported modules.

Your code knows the directory.  My version would search in your
package __path__ for it.  But you only have one wininst.exe per
package, right?  And maybe putting it on package path is a feature.

JimA



From just@letterror.com  Fri Dec 20 20:19:52 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 20 Dec 2002 21:19:52 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <u1h8zd6q.fsf@python.net>
Message-ID: <r01050400-1022-6D8376CC145811D7A7C4003065D5E7E4@[10.0.0.1]>

Thomas Heller wrote:

> Looks ok to me, although I would prefer to write
> 
>         path = os.sep.join(__name__.split(".") + ["wininst.exe"])
> or
>         path = os.path.join(*__name__.split(".") + ["wininst.exe"])

Much better. I think I'd prefer the first, mostly because os.path.join()
might do more magic than needed.

I think it's time to pin down the next extensions to the importer
protocol. How about the next speclet:

    bool = loader.is_package(fullname)
    code = loader.get_code(fullname)
    str = loader.get_source(fullname)

(is_package and get_code are needed for modulefinder-like tools,
get_source is needed for linecache-like tools.)

All three should raise ImportError if the module isn't available.
get_code and get_source should return None if there's no code or source
associated with the module, respectively. "No code" means it's a builtin
module or an extension. "No source" means just that (eg. for zipimport
it would mean "I've only got a .pyc or .pyo file for you").

If the importer does support get_source(), yet doesn't have a code
object readily available, get_code() should compile the source by itself
and return the code. This is to make life easier for the caller: if it
only needs the code object it shouldn't need to also check get_source()
and deal with that.

They should be optional, yet strongly recommended for general purpose
hooks. There's no need to support them for hooks that are specific to a
deliverable app; for example zipimport should have them, but if Thomas
writes a dedicated hook for the apps built by py2exe, he should be free
to leave them out.

Btw. would it be better if mod.__importer__ was named mod.__loader__?
This is closer to the truth, yet I fear that __loader__ is too generic.

> > PS: please take care to not Cc to peps@python.org any more in this
> > thread, I think I've done enough harm ;-)
> 
> Sorry for that, I noticed it as it was already too late.

No need to be sorry, it was my fault...

Just


From jim@interet.com  Fri Dec 20 20:41:29 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 20 Dec 2002 15:41:29 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
References: <r01050400-1022-2E4EDCBE144E11D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <3E038079.3040607@interet.com>

Just van Rossum wrote:

> One more thought: finding modules and finding data files is not the same
> thing. Modules can be anywhere and must be accessible to anyone, but
> data files belong to specific modules. Searching the entire sys.path for
> a data file seems way too broad; there's got to be some information
> available as to where the data is. Eg. I doubt it's a *feature* that
> pdb.doc may also be found in site-packages.

I don't think that is a problem for packages, because
imp.find_module() does not accept dotted names now.  To import
P.M you must currently import P, and then use P.__path__ as the path
in a call to imp.find_module().
Presumably the package author knows her package path, and
can use that in imp.find_module().  We aren't looking for
data files in Python std lib, right?

Hmmm.  You have a point if the data is specific to a module and
not to a package.  It seems we are forced to use __file__, the
only way to know the module's source.  The __file__ can be
    /usr/lib/python22/lib/os.pyc
    /usr/lib/python/python22.zip/os.pyc
so if we strip off "os.pyc" the rest can be used as the path.

Hmmm.  And if __file__ is None???

JimA



From theller@python.net  Fri Dec 20 20:52:41 2002
From: theller@python.net (Thomas Heller)
Date: 20 Dec 2002 21:52:41 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <r01050400-1022-6D8376CC145811D7A7C4003065D5E7E4@[10.0.0.1]>
References: <r01050400-1022-6D8376CC145811D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <lm2kza3q.fsf@python.net>

Just van Rossum <just@letterror.com> writes:

> Thomas Heller wrote:
> 
> > Looks ok to me, although I would prefer to write
> > 
> >         path = os.sep.join(__name__.split(".") + ["wininst.exe"])
> > or
> >         path = os.path.join(*__name__.split(".") + ["wininst.exe"])
> 
> Much better. I think I'd prefer the first, mostly because os.path.join()
> might do more magic than needed.

Me too. Also because I wasn't sure first if I had to write
         path = os.path.join(*l1 + l2)
or
         path = os.path.join(*(l1 + l2))

> 
> I think it's time to pin down the next extensions to the importer
> protocol. How about the next speclet:
> 
>     bool = loader.is_package(fullname)
>     code = loader.get_code(fullname)
>     str = loader.get_source(fullname)
> 
> (is_package and get_code are needed for modulefinder-like tools,
> get_source is needed for linecache-like tools.)

I like this much better than Jim's proposal, although I cannot think
too deep currently about this.

> 
> All three should raise ImportError if the module isn't available.
> get_code and get_source should return None if there's no code or source
> associated with the module, respectively. "No code" means it's a builtin
> module or an extension. "No source" means just that (eg. for zipimport
> it would mean "I've only got a .pyc or .pyo file for you").
> 
> If the importer does support get_source(), yet doesn't have a code
> object readily available, get_code() should compile the source by itself
> and return the code. This is to make life easier for the caller: if it
> only needs the code object it shouldn't need to also check get_source()
> and deal with that.
> 
> They should be optional, yet strongly recommended for general purpose
> hooks. There's no need to support them for hooks that are specific to a
> deliverable app; for example zipimport should have them, but if Thomas
> writes a dedicated hook for the apps built by py2exe, he should be free
> to leave them out.
> 
> Btw. would it be better if mod.__importer__ was named mod.__loader__?

Yes.

> This is closer to the truth, yet I fear that __loader__ is too generic.

Before I can give better comments, I have to read the PEP it seems.

Thomas



From jim@interet.com  Fri Dec 20 21:02:09 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 20 Dec 2002 16:02:09 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
References: <r01050400-1022-6B419BC4144811D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <3E038551.6030502@interet.com>

Just van Rossum wrote:

> I didn't change anything there and I sure don't break any code, because
> there's hardly any code out there that expects imp.find_module() to work
> for zip files <0.5 wink>.

Huh?  In my implementation imp.find_module() imp.load_module()
work for zip files.  You mean they don't work to provide access
to data files, right?

>>      find_module(name [, path [, suffixes]])

> I'm not sure if I like this:

Me either, it was just an idea to solve a prominent problem.

> I'll think about this some more, but I'm not yet convinced.

Me too.

JimA



From just@letterror.com  Fri Dec 20 22:19:30 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 20 Dec 2002 23:19:30 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <3E038551.6030502@interet.com>
Message-ID: <r01050400-1022-2148C4FE146911D7A7C4003065D5E7E4@[10.0.0.1]>

James C. Ahlstrom wrote:

> > I didn't change anything there and I sure don't break any code,
> > because there's hardly any code out there that expects
> > imp.find_module() to work for zip files <0.5 wink>.
> 
> Huh?  In my implementation imp.find_module() imp.load_module()
> work for zip files. 

I know, but this is only possible when the import extension is hard
coded in import.c, and lifting the need for hard coded hooks in import.c
was the entire point of my exercise. This comes at the cost of it not
working with imp.find_module().

> You mean they don't work to provide access to data files, right?

Misunderstanding, I meant what I said. In my implementation
imp.find_module() *doesn't* work for zip files, but I don't think that's
such a big deal as zipimport is a new thing, or rather, the new import
hooks are, erm, new. You've hit a weak spot in my scheme though, and
perhaps the PEP should make that clearer. Nevertheless I see no decent
way to integrate the importer protocol with imp.find_module/load_module.

The way I see it (as I tried to say in the PEP) is that
imp.find_module/load_module interfaces the builtin import mechanism as
it is in Python 2.2 and before (builtin, frozen and plain directory
imports, that's it). imp.load_module() is a primitive dispatcher that
switches on the filetype enum field of a filedescr struct. What is
needed with the new hooks is that the load_module() of a importer/loader
object be called. This is a modernization (OO-ification) of the same
idea, but obviously not compatible with the current imp module's world
view. Hence my proposal to add a new function called imp.find_module2(),
which also takes into account that the importer protocol needs the
*full* module name -- another (even more unfixable, I think)
incompatibility with imp.find_module().

(Btw. the only uses in the standard library of imp.find_module or
imp.load_module are in ihooks.py imputil.py, pyclbr.py and pydoc.py.
It's not like it's an insanely popular API, and I stick with my claim
that I didn't break anything ;-)

Just


From lists@morpheus.demon.co.uk  Fri Dec 20 23:32:26 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Fri, 20 Dec 2002 23:32:26 +0000
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
References: <r01050400-1022-6B419BC4144811D7A7C4003065D5E7E4@[10.0.0.1]> <znr0zf2y.fsf@python.net>
 <3E037BFE.7020509@interet.com>
Message-ID: <n2m-g.isxow9kl.fsf@morpheus.demon.co.uk>

"James C. Ahlstrom" <jim@interet.com> writes:

> It would look like this:
>
>    suffix = [(".exe", "rb", 0)]
>    file, path, descr = imp.find_module("wininst", __path__, suffix)
>    data = file.read()
>    file.close()
>    return data

It's an interesting idea, but like others, I think that it's not going
to work right in practice. The main problem is that you don't explain
how to implement this version of imp.find_module in terms of the
importer protocol defined in the hooks PEP.

> Note that for a zip file, __file__ is something like
>       C:/Python/lib/zarchive.zip/subdir/mymod.pyc
> and nothing is going to make code using __file__ Just Work.

That's not true. Under your patch, sure that's what __file__
is. Offhand, I'm not sure what Just's zipimporter code puts in there,
and I certainly wouldn't guarantee it for an arbitrary implementation
of zip imports. Frozen and builtin modules don't have usable __file__
attributes, and for something like a hook loading modules from a
database, there is no meaningful __file__ value.

> Well, maybe you could strip the last component "mymod.pyc" and
> use it as __path__ in a call to imp.find_module().  Not too
> hard, and it works with files and zip archives.  But maybe not
> with future ftp:// imported modules.

And of course only with zip archives if you make imp.find_module()
support that usage. This is a circular argument.

> Your code knows the directory.  My version would search in your
> package __path__ for it.  But you only have one wininst.exe per
> package, right?  And maybe putting it on package path is a feature.

That's what the get_data(name) method Just is proposing is supposed to
address. The only difficulty with it is pinning down what the "name"
argument means. At the lowest level, it's an arbitrary cookie which
identifies a chunk of data. The trick is to avoid making that cookie
*too* unrelated to how things work in the filesystem...

Paul.
-- 
This signature intentionally left blank


From Jack.Jansen@oratrix.com  Sat Dec 21 13:57:24 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Sat, 21 Dec 2002 14:57:24 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <r01050400-1022-6D8376CC145811D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <22166EBA-14EC-11D7-ACA8-000A27B19B96@oratrix.com>

On vrijdag, dec 20, 2002, at 21:19 Europe/Amsterdam, Just van Rossum 
wrote:

> Thomas Heller wrote:
>
>> Looks ok to me, although I would prefer to write
>>
>>         path = os.sep.join(__name__.split(".") + ["wininst.exe"])
>> or
>>         path = os.path.join(*__name__.split(".") + ["wininst.exe"])
>
> Much better. I think I'd prefer the first, mostly because 
> os.path.join()
> might do more magic than needed.

But that magic would actually be needed for MacOS9 pathnames. 
os.path.join(*['foo', 'bar']) will correctly return ':foo:bar', whereas 
os.sep.join will return 'foo:bar', which is wrong.

Not that we should care all that much anymroe about MacOS9 pathnames, 
but still...
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From just@letterror.com  Sat Dec 21 14:38:39 2002
From: just@letterror.com (Just van Rossum)
Date: Sat, 21 Dec 2002 15:38:39 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <n2m-g.isxow9kl.fsf@morpheus.demon.co.uk>
Message-ID: <r01050400-1022-EB44A3B414F111D7A7C4003065D5E7E4@[10.0.0.1]>

Paul Moore wrote:

> > Note that for a zip file, __file__ is something like
> >       C:/Python/lib/zarchive.zip/subdir/mymod.pyc
> > and nothing is going to make code using __file__ Just Work.
> 
> That's not true. Under your patch, sure that's what __file__
> is. Offhand, I'm not sure what Just's zipimporter code puts in there,

It actually does the same thing as Jim's patch regarding __file__. But
he's right in both cases: using __file__ is will not "Just Work" ;-)

> and I certainly wouldn't guarantee it for an arbitrary implementation
> of zip imports. Frozen and builtin modules don't have usable __file__
> attributes, and for something like a hook loading modules from a
> database, there is no meaningful __file__ value.

Indeed.

[ ... ]
> That's what the get_data(name) method Just is proposing is supposed to
> address. The only difficulty with it is pinning down what the "name"
> argument means. At the lowest level, it's an arbitrary cookie which
> identifies a chunk of data. The trick is to avoid making that cookie
> *too* unrelated to how things work in the filesystem...

I just wrote this to Paul in private mail:

    The 'name' argument of i.get_data(name) should be seen as a
    'cookie', meaning the importer protocol doesn't prescribe any
    semantics for it. However, for importer objects that have some
    file system-like properties (for example zipimporter) it is
    recommended to use os.sep as a separator character to specify a 
    (possibly virtual) directories hierarchy. For example if the 
    importer allows access to a module's source code via 
    i.get_data(name), the 'name' argument should be constructed like 
    this:

        name = mod.__name__.replace(".", os.sep) + ".py"

    Note that this is not the recommended way to retrieve source code,
    the (optional) method i.get_source(fullname) is more general, as
    it doesn't imply *any* file-system-like characteristics.

But in the light of Jack's remark regarding MacOS<X pathnames it might
be better to stick with '/' instead of os.sep. This is not a real file
system path, so it seems odd to enforce platform-specific path
semantics. From Jack's post:

> > Much better. I think I'd prefer the first, mostly because
> > os.path.join() might do more magic than needed.
> 
> But that magic would actually be needed for MacOS9 pathnames.
> os.path.join(*['foo', 'bar']) will correctly return ':foo:bar',
> whereas os.sep.join will return 'foo:bar', which is wrong.

"needed" and "wrong" are highly questionalble in the context of
importer.get_data()...

Just


From nas@python.ca  Sun Dec 22 02:25:52 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 21 Dec 2002 18:25:52 -0800
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>
References: <r01050400-1022-E143AA2D135311D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <20021222022552.GA12082@glacier.arctrix.com>

Just van Rossum wrote:
>     This PEP proposes to add a new set of import hooks that offer better
>     customization of the Python import mechanism.  Contrary to the
>     current __import__ hook, a new-style hook can be injected into the
>     existing scheme, allowing for a finer grained control of how modules
>     are found and how they are loaded.

I only had time to skim this PEP.  Currently I am using import hooks to
load PTL modules (like .py modules but they have a different extension
and need a different compiler).  Most of the discussion has focused on
loading .py modules from places other than filesystems.  I was afraid my
use case would not get addressed (iu.py doesn't really work, ihooks.py
does).  It looks like the hooks proposed by this PEP are sufficent for
my needs.

This PEP addresses a real problem, IMHO.  The current __import__ hook is
hard to use.  There is too much complicated package import logic that
must be reimplemented.

  Neil


From just@letterror.com  Sun Dec 22 18:07:52 2002
From: just@letterror.com (Just van Rossum)
Date: Sun, 22 Dec 2002 19:07:52 +0100
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <20021222022552.GA12082@glacier.arctrix.com>
Message-ID: <r01050400-1022-4E2B13E315D811D7A7C4003065D5E7E4@[10.0.0.1]>

Neil Schemenauer wrote:

> I only had time to skim this PEP.  Currently I am using import hooks
> to load PTL modules (like .py modules but they have a different
> extension and need a different compiler).  Most of the discussion has
> focused on loading .py modules from places other than filesystems.  I
> was afraid my use case would not get addressed (iu.py doesn't really
> work, ihooks.py does).  It looks like the hooks proposed by this PEP
> are sufficent for my needs.

Interesting use case! I wish I had known about it earlier. Can you point
me to your code using ihooks, if it's available?

I'm actually not so sure the new hook mechanism solves all your problems
as well as it could/should. I assume these special modules simply live
on sys.path, intermixed with "regular" modules? This would mean you
specifically need to extend the builtin sys.path/pkg.__path__ find/load
logic, which is something the PEP doesn't cater for (yet). If the
special modules live in a special folder, things are easier. Also:
special modules will not work when packaged in a zip file (zipimport
doesn't come *close* to transparently replacing the file system). If you
compile to straight .py[co] (as I assume) and put those in the zip
archive, you should be fine, though.

This also touches on the modulefinder-like-tools-issue. Your hook should
ideally support the get_code(fullname) method, so that packaging tools
will be able to deal with it more or less transparently. Obviously the
hook need to be installed when running modulefinder-like-tools. (Hmm,
this suggests that an external hook registry might be handy, perhaps a
*.pth-like trick...)

Just


From nas@python.ca  Mon Dec 23 06:46:12 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sun, 22 Dec 2002 22:46:12 -0800
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for PEP #)
In-Reply-To: <r01050400-1022-4E2B13E315D811D7A7C4003065D5E7E4@[10.0.0.1]>
References: <20021222022552.GA12082@glacier.arctrix.com> <r01050400-1022-4E2B13E315D811D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <20021223064612.GA14325@glacier.arctrix.com>

[I'm on a really slow link so this won't be detailed :-(]

> Interesting use case! I wish I had known about it earlier. Can you
> point me to your code using ihooks, if it's available?

It's part of Quixote (google knows where).  The module is ptl_import.py.

> I'm actually not so sure the new hook mechanism solves all your
> problems as well as it could/should. I assume these special modules
> simply live on sys.path, intermixed with "regular" modules? 

That's the way the current hooks work.  It probably would be okay if
certain directorys could contain only .ptl modules.  I'm not sure our
users would like that incompatibilty.  Obviously it would be nice if the
new hooks could easily support the current behavior.

  Neil


From bbum@codefab.com  Mon Dec 23 16:58:49 2002
From: bbum@codefab.com (Bill Bumgarner)
Date: Mon, 23 Dec 2002 11:58:49 -0500
Subject: [Python-Dev] ANN: PyObjC 0.8
Message-ID: <CE9CE0DA-1697-11D7-9679-000393877AE4@codefab.com>

PyObjC version 0.8 is now available for download.

See...

     http://pyobjc.sourceforge.net/

... for more information.

The installer package includes a Project Builder project template for  
easily creating new Cocoa-Python projects and a slew of examples.   
Alternatively, the developer can choose to develop applications without  
using Apple's developer tools. Full support for Interface Builder  
documents is included.

The PyObjC Python module provides a feature complete bridge between the  
Python programming language and Objective-C. With PyObjC, it is  
possible to transparently message Objective-C objects from Python and  
Python objects from Objective-C. Objective-C objects can be subclassed  
from within Python.  Support is also included for using the various  
functions, enumerated types, and global variables found throughout the  
Objective-C framework.

References to python dictionaries pass into Objective-C act like a well  
behaved member of the NSDictionary class cluster. Likewise, Python List  
references behave like a member of the NSArray class cluster.
Strings are transparently converted to the native type.  NSArray and  
NSDictionary references passed into Python behave like native Python  
lists and dictionaries; including the ability to enumerate either using  
the 'in' operator and to slice NSArray and NSMutableArray instances.

PyObjC fully supports creating full featured Cocoa applications written  
in pure Python.  There are aspects of PyObjC that are more powerful  
than Cocoa in pure Obj-C (the ability to automatically define  
classes/outlets based on the contents of a NIB file, for example).

PyObjC requires OS X 10.2 or greater.  10.1 support is possible and  
will likely happen soon-- contact me if you need 10.1 support and are  
willing to do a bit of grunt work to generate the appropriate files  
(easy to do-- just need a 10.1 development machine).

The installer package is designed to work with the built-in Python  
provided in OS X 10.2.   Source is included on the disk image and the  
pyobjc module works with Python 2.2 or greater as installed directly  
from the Python Source, with the MacPython packages, and with the Fink  
build of python.

PyObjC also provides an awesome environment for exploring frameworks.    
The following transcript was copied out of a Terminal window-- it is an  
example of working directly with the Objective-C runtime from Python  
within the terminal.

 >>> from objc import *
 >>> from Foundation import *
 >>> b =  
NSBundle.bundleWithPath_("/System/Library/PrivateFrameworks/ 
PBXCore.framework")
 >>> b.principalClass()
<objective-c class PBXProducerTarget at 0xa9030034>
 >>> NSBundle.searchPathsForSupportFilesWithSubpath_("FooBar")
(
     "/Volumes/Data/Users/bbum/Developer/ProjectBuilder Extras/FooBar",
     "/Network/Developer/ProjectBuilder Extras/FooBar",
     "/Developer/ProjectBuilder Extras/FooBar"
)
 >>> PBXProject = lookUpClass("PBXProject")
 >>> p =  
PBXProject.projectWithFile_("/Developer/Examples/AppKit/TextEdit/ 
TextEdit.pbproj")
 >>> p.targets()
(<PBXApplicationTarget:0x00a76830:6FE2093EFE93D5F211CA2CEA:name='TextEdi 
t'>)
 >>> p.buildStyles()
(
      
<PBXBuildStyle:0x00a77d30:011ADBA3FF9FD52E11CA0C5D:name='Development':bu 
ildSettings.count=2>,
      
<PBXBuildStyle:0x00a82180:011ADBA4FF9FD52E11CA0C5D:name='Deployment':bui 
ldSettings.count=1>
)

The version # of 0.8 indicates that documentation is lacking, unit  
tests are incomplete and there may be a bug or two lurking within.  In  
practice, the module has proven to be very stable and to allow for the  
development of extremely complex Cocoa projects in pure Python or a mix  
of Python and Objective-C.



From exarkun@meson.dyndns.org  Tue Dec 24 07:09:41 2002
From: exarkun@meson.dyndns.org (Jp Calderone)
Date: Tue, 24 Dec 2002 02:09:41 -0500
Subject: [Python-Dev] inet_ntop/inet_pton
Message-ID: <20021224070941.GA26312@meson.dyndns.org>

--ZPt4rx8FFjLCG7dd
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

  I'm curious if 2.3 is going to make these two functions available in the
socket module.  There seems to be some support for them (or at least an
attempt at functionality on platforms that don't support them) in
socketmodule.c, but I don't see a Python interface for them.  Is someone
working on this?  Should I work up a patch myself? :)

  Jp
--
 12:00am up 8 days, 9:45, 3 users, load average: 0.05, 0.06, 0.01

--ZPt4rx8FFjLCG7dd
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD8DBQE+CAg1edcO2BJA+4YRAubLAKDGav4gohL771bVdjOR0wzk3kL3SgCeJH0o
0hBl0ANf1vzvWl24VHc8PjM=
=RcFu
-----END PGP SIGNATURE-----

--ZPt4rx8FFjLCG7dd--


From guido@python.org  Tue Dec 24 19:30:34 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 24 Dec 2002 14:30:34 -0500
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
Message-ID: <200212241930.gBOJUYD10795@odiug.zope.com>

I expect that I'll have more time for the Python 2.3 release the rest
of this year, so I will do my darndest to do an alpha release before
the year is over.  I've partially updated PEP 283 (the release
schedule), though there are still a lot of things to be done that
aren't there.  I'll get to these on Thursday.

I've read PEP 302, and while there are some things left to discuss, I
would really like to see Just's code checked in for the alpha release,
so that it gets widespread testing.  Just, are you up for that?

Happy holidays everybody!

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Tue Dec 24 20:48:12 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 24 Dec 2002 21:48:12 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
In-Reply-To: <200212241930.gBOJUYD10795@odiug.zope.com>
Message-ID: <r01050400-1022-12EA2664178111D7A7C4003065D5E7E4@[10.0.0.1]>

Guido van Rossum wrote:

> I've read PEP 302, and while there are some things left to discuss, I
> would really like to see Just's code checked in for the alpha release,
> so that it gets widespread testing.  Just, are you up for that?

Sure ;-) I'll wait a few days to see if there are any objections or
showstoppers.

I updated the patch yesterday, there was a fairly large change in
zipimport.c. If it were in CVS, this would have been the log msg:

   general:
   - incorporated patch from Paul Moore that adds a default zip
   archive path to sys.path on Windows (it was already there on
   unix). Thanks Paul!
   
   import.c:
   - matches latest version of PEP 302 (rev 1.3), regarding the
   new 'path' argument of importer.find_module(fullname, path=None)
   
   zipimporter.c
   - removed the subdir feature, which allowed the path to the
   zip archive to be extended with a subdirectory. PEP 273
   stated this was needed for package support (and only for
   that). However, with the new import hooks this is no longer
   true: a path item containing the plain zip archive path can
   also deal with submodules (find_module receives the full
   module name after all). Therefore a pkg.__path__ from a
   package loaded from a zip archive will contain the *plain*
   zip archive path.
   - as a consequence I could simplify and clean up lots of
   things (esp. zipimporter_init: eg. it no longer needs to
   check sys.path_importer_cache; yay). Getting rid of the
   zipimporter.prefix attribute altogether helped a lot in
   other places.
   - this change additionally enabled me to get rid of the
   restriction that zip paths must end in .ZIP or .zip; any
   extension (or even no extension) will now work.
   - implemented all the (optional) extensions of the Importer
   Protocol that the latest version of PEP 302 defines.

Merry Christmas everyone!

Just


From pedronis@bluewin.ch  Tue Dec 24 21:09:32 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 24 Dec 2002 22:09:32 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
References: <r01050400-1022-12EA2664178111D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <023701c2ab90$c1736240$6d94fea9@newmexico>

>    zipimporter.c
>    - removed the subdir feature, which allowed the path to the
>    zip archive to be extended with a subdirectory. PEP 273
>    stated this was needed for package support (and only for
>    that). However, with the new import hooks this is no longer
>    true: a path item containing the plain zip archive path can
>    also deal with submodules (find_module receives the full
>    module name after all). Therefore a pkg.__path__ from a
>    package loaded from a zip archive will contain the *plain*
>    zip archive path.
>    - as a consequence I could simplify and clean up lots of
>    things (esp. zipimporter_init: eg. it no longer needs to
>    check sys.path_importer_cache; yay). Getting rid of the
>    zipimporter.prefix attribute altogether helped a lot in
>    other places.
>    - this change additionally enabled me to get rid of the
>    restriction that zip paths must end in .ZIP or .zip; any
>    extension (or even no extension) will now work.

will not this break __path__ manipulations?

regards.


From pedronis@bluewin.ch  Tue Dec 24 22:45:40 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Tue, 24 Dec 2002 23:45:40 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
References: <r01050400-1022-12EA2664178111D7A7C4003065D5E7E4@[10.0.0.1]> <023701c2ab90$c1736240$6d94fea9@newmexico>
Message-ID: <02a801c2ab9e$2f6ab520$6d94fea9@newmexico>

From: "Samuele Pedroni" <pedronis@bluewin.ch>
> >    zipimporter.c
> >    - removed the subdir feature, which allowed the path to the
> >    zip archive to be extended with a subdirectory. PEP 273
> >    stated this was needed for package support (and only for
> >    that). However, with the new import hooks this is no longer
> >    true: a path item containing the plain zip archive path can
> >    also deal with submodules (find_module receives the full
> >    module name after all). Therefore a pkg.__path__ from a
> >    package loaded from a zip archive will contain the *plain*
> >    zip archive path.
> >    - as a consequence I could simplify and clean up lots of
> >    things (esp. zipimporter_init: eg. it no longer needs to
> >    check sys.path_importer_cache; yay). Getting rid of the
> >    zipimporter.prefix attribute altogether helped a lot in
> >    other places.
> >    - this change additionally enabled me to get rid of the
> >    restriction that zip paths must end in .ZIP or .zip; any
> >    extension (or even no extension) will now work.
>
> will not this break __path__ manipulations?
>

further this change is backward incompatible with what is allowed by Jython
2.1,

it was considered a feature to be able to put

path/to/a.zip/python

in sys.path, so that the a.b package would be looked up under

path/to/a.zip/python/a/b.

regards.




From just@letterror.com  Wed Dec 25 09:03:45 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 25 Dec 2002 10:03:45 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
In-Reply-To: <02a801c2ab9e$2f6ab520$6d94fea9@newmexico>
Message-ID: <r01050400-1022-C8E9667517E711D7A7C4003065D5E7E4@[10.0.0.1]>

From: "Samuele Pedroni" <pedronis@bluewin.ch>
> >    zipimporter.c
> >    - removed the subdir feature, which allowed the path to the
> >    zip archive to be extended with a subdirectory. PEP 273
> >    stated this was needed for package support (and only for
> >    that). However, with the new import hooks this is no longer
> >    true: a path item containing the plain zip archive path can
> >    also deal with submodules (find_module receives the full
> >    module name after all). Therefore a pkg.__path__ from a
> >    package loaded from a zip archive will contain the *plain*
> >    zip archive path.
> >    - as a consequence I could simplify and clean up lots of
> >    things (esp. zipimporter_init: eg. it no longer needs to
> >    check sys.path_importer_cache; yay). Getting rid of the
> >    zipimporter.prefix attribute altogether helped a lot in
> >    other places.
> >    - this change additionally enabled me to get rid of the
> >    restriction that zip paths must end in .ZIP or .zip; any
> >    extension (or even no extension) will now work.
>
> will not this break __path__ manipulations?

If the particular manipulation did work for zip files at all before, yes
:-(. (It wouldn't have worked with a Zip archive that was packed by a
freeze-like tool, unless the *results* of the manipulations were
explicitly flattened during packaging.)

> further this change is backward incompatible with what is allowed by
> Jython 2.1,
> 
> it was considered a feature to be able to put
> 
> path/to/a.zip/python
> 
> in sys.path, so that the a.b package would be looked up under
> 
> path/to/a.zip/python/a/b.

PEP 273 doesn't document it as such; it only says it's needed for
package imports. Also, to be honest, my implementation had some issues
with that usage: it would look for the plain .zip archive in
sys.path_importer_cache, which would obviously not be found, causing the
zip file index to be read again for every package directory. The only
solution I thought of that could solve that is for the zipimporter
object to *add* entries to sys.path_importer_cache itself, and I found
it bad enough already that it _read_ from the cache itself in the
previous version. It's a messy feature :-(.

I personally don't care about this feature; it's easy enough to package
the archive so that it's not needed.

Regarding the __path__ manipulations: this assumes file-system
properties and can't work for importers in *general* and it feels like a
hack to specially allow it for Zip archives (it definitely was a hack in
my implementation, therefore I'm happy to get rid of it ;-). In many
other respects Zip archives also won't be able to be compatible with a
real file system anyway, eg. why should __path__ manipulations work and
not __file__ manipulations? (Now if we had a virtual file system with
Zip file support, things would be different!)

I still think that __path__ manipulations are evil, as would be
module-specific sys.path manipulation. To me, sys.path is the domain of
*applications*, which implies that pkg.__path__ should be left alone
also (at least by the package itself). It seems Guido is going the
opposite direction with pkgutil.py :-(.

Just


From pedronis@bluewin.ch  Wed Dec 25 12:38:33 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Wed, 25 Dec 2002 13:38:33 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
References: <r01050400-1022-C8E9667517E711D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <001801c2ac12$89f4b620$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> I still think that __path__ manipulations are evil, as would be
> module-specific sys.path manipulation. To me, sys.path is the domain of
> *applications*, which implies that pkg.__path__ should be left alone
> also (at least by the package itself). It seems Guido is going the
> opposite direction with pkgutil.py :-(.

Indeed it seemed that there was consensus to want them (messy, hackish or
whatever they may seem).

Honestly non considering the internals, the new __path__ conveys _less_
information that it could. That's the crucial point in my eyes.

I'm -1 on the changes.

regards.



From just@letterror.com  Wed Dec 25 14:16:39 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 25 Dec 2002 15:16:39 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
In-Reply-To: <001801c2ac12$89f4b620$6d94fea9@newmexico>
Message-ID: <r01050400-1022-7F6A7106181311D7A7C4003065D5E7E4@[10.0.0.1]>

Samuele Pedroni wrote:

> Indeed it seemed that there was consensus to want [__path__
> manipulations] (messy, hackish or whatever they may seem).

But, as I wrote, it *can't* work with importers in general. It can be
made to work with specific importers, such as zipimporter, but I don't
see the point (I'll rant about that in a second ;-).

> Honestly non considering the internals, the new __path__ conveys
> _less_ information that it could. That's the crucial point in my
> eyes.

For zipimporter, yes, you can see it that way. I just think it conveys
*different* information.

> I'm -1 on the changes.

:-(

I still have to see an example of a pkg.__path__ manipulation in which
the intentions couldn't be solved in a different way, for example like
how os.py imports platform-specific features. If os.py would have done
that by munging sys.path everybody would have been appalled.

The more I think about it, the more I think we should deprecate __path__
altogether. The path import mechanism code could just as well look for
"package/submodule.py" on sys.path (which would in fact make pkgutils.py
redundant as the feature it currently implements would then be standard
behavior). That's a bit extreme and is perhaps not even doable in a 100%
b/w compatible way, yet I do believe __path__ was a design mistake.
http://www.python.org/doc/essays/packages.html doesn't explain the
rationale as to why this particular solution was chosen; I _suspect_ it
was simply an easy solution that left much of the lower level import
logic unchanged. If that's true, then being able to modify pkg.__path__
is merely a side effect of the chosen solution. I could be wrong,
though...

Repackaging applications for use by end users (with freeze, py2exe, etc)
should be easy and straightforward. Manipulating pkg.__path__ makes this
harder than neccesary and that alone should be reason enough to strongly
discourage it.

Just


From pedronis@bluewin.ch  Wed Dec 25 16:06:43 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Wed, 25 Dec 2002 17:06:43 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
References: <r01050400-1022-7F6A7106181311D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <00ec01c2ac2f$9e2720c0$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>

> Samuele Pedroni wrote:
>
> > Indeed it seemed that there was consensus to want [__path__
> > manipulations] (messy, hackish or whatever they may seem).
>
> But, as I wrote, it *can't* work with importers in general. It can be
> made to work with specific importers, such as zipimporter, but I don't
> see the point (I'll rant about that in a second ;-).

but such importers would better go in meta_path.

Btw, reading PEP 302 it seems that when dealing with a package,
first *all* importers in sys.meta_path called with

fullname and __path__ of the package.

For symmetry (as long as we keep __path__) I think the import mechanism would
have to look
for a list in __meta__path__ (or maybe just a single value in
__meta_importer__):

if it's absent, thing should work as described above,

if it's empty, sys.meta_path should be ignored,

otherwise it's content should be used instead of sys.meta_path and __path__
should be ignored up to passing it to the meta importer(s).

Basically this would mean that the contents of __path__ should be
interpreted/used only by the meta importer.

regards






From just@letterror.com  Wed Dec 25 21:42:45 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 25 Dec 2002 22:42:45 +0100
Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31)
In-Reply-To: <00ec01c2ac2f$9e2720c0$6d94fea9@newmexico>
Message-ID: <r01050400-1022-D3720244185111D7A7C4003065D5E7E4@[10.0.0.1]>

Samuele Pedroni wrote:

> From: "Just van Rossum" <just@letterror.com>
> 
> > But, as I wrote, it *can't* work with importers in general. It can
> > be made to work with specific importers, such as zipimporter, but I
> > don't see the point (I'll rant about that in a second ;-).
> 
> but such importers would better go in meta_path.

Yes, but that doesn't make modifying __path__ less fragile ;-)

> Btw, reading PEP 302 it seems that when dealing with a package, first
> *all* importers in sys.meta_path called with
> 
> fullname and __path__ of the package.

Correct.

> For symmetry (as long as we keep __path__) I think the import
> mechanism would have to look for a list in __meta__path__ (or maybe
> just a single value in __meta_importer__):

Hm, you may have a point there.

However, this makes life yet a bit harder for tools that only *find*
modules, and not load them (eg. modulefinder.py). This is already a
problem (even without the new hooks, but less so) with __path__:
currently a modulefinder (let's use that term for a generic non-loading
module finding tool) has to calculate what __path__ *would* be, were it
actually loaded, and pass that as the second argument to
imp.find_module(). For regular file system imports this is not such a
big deal, but for importers that have no notion of a file system, yet
use pkg.__path__, it is. The PEP proposes an optional method for loader
objects named get_path(), that would return what __path__ would be set
to were the package actually loaded. I'm not all that excited about the
neccessity for this... pkg.__meta_path__ would make things yet more
complicated (does it need to be passed to imp.find_module as well?).
What you call __meta_importer__ might fit into the scheme, perhaps just
as __importer__ (and rename what the PEP now defines as __importer__ to
__loader__, which seems a good idea anyway). I think iu.py does
something similar: it adds a __importsub__ symbol to packages, which is
a function to import submodules.

However, this all leads to a narrower search path for submodules. I want
it to be *wider*.

Back to __path__: about the only reason I can think for having it in the
first place (besides being the "I'm a package" flag) is optimization: it
avoids some stat calls by restricting the lookup to a specific
directory. Yet there's the desire to import submodules from other
places. Eg: loading .pyc modules from a zip archive, yet have
dynamically loaded extensions come from the file system. Extension
modules as submodules of frozen packages is currently not possible
without deeper trickery: __path__ is a string in that case, so modifying
it wouldn't be possible, let alone solve the problem.

Let's assume for a moment we want to deprecate __path__. (Hm, I tried --
in my mind <wink> -- to also get rid of the "I'm a package" flag, but
this is not realistic as long as we have implicit relative imports.)
Submodules would be found on sys.path by constructing a relative path
like "package/submodule.py".

Here's a sample directory structure:

    dir1/spam/__init__.py
    dir1/spam/ham.py
    
    dir2/spam/eggs.pyd

dir1 and dir2 are items on sys.path. "import spam.eggs" would first load
"spam" from dir1/spam/__init__.py and then load "spam.eggs" from
dir2/spam/eggs.pyd. It appears as if this is more or less what
pkgutils.py tries to accomplish, yet brings the control of this feature
back to the domain of the application, where it belongs.

It also makes life easier for modulefinders: imp.find_module() could now
be made to accept fully qualified module names (only when no 'path'
argument is given). It would *also* allow me to get rid of the second
argument of the find_module() method of the importer protocol as well as
of the proposed imp.find_module2() function. It makes a lot of things
simpler...

Yet one more thing this scheme enables: extension modules as submodules
in a multiplatform library setup (ie. different platforms using the same
library on a server).

Finding and loading of submodules would be completely decoupled from the
parent module; parent and child could be imported by completely
unrelated importers. The latter is already a *feature* of sys.meta_path
(which explains why I'm not thrilled about Samuele's proposal ;-) and I
would love for this to be extended to file system imports.

Just


From Jack.Jansen@oratrix.com  Wed Dec 25 22:00:43 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Wed, 25 Dec 2002 23:00:43 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
In-Reply-To: <r01050400-1022-12EA2664178111D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <5085E098-1854-11D7-BC72-000A27B19B96@oratrix.com>

On dinsdag, dec 24, 2002, at 21:48 Europe/Amsterdam, Just van Rossum 
wrote:

> Guido van Rossum wrote:
>
>> I've read PEP 302, and while there are some things left to discuss, I
>> would really like to see Just's code checked in for the alpha release,
>> so that it gets widespread testing.  Just, are you up for that?
>
> Sure ;-) I'll wait a few days to see if there are any objections or
> showstoppers.

Please don't wait too long, as I have no reasonable way to apply the 
patch on MacOS9, and I can imagine that MacPython-OS9 might encounter 
problems, with its own hooks in import.c. If you don't want to check it 
in on the trunk please do so on a branch.

Or did you test MacPython-OS9 interaction yourself?
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From pedronis@bluewin.ch  Wed Dec 25 22:02:08 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Wed, 25 Dec 2002 23:02:08 +0100
Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31)
References: <r01050400-1022-D3720244185111D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <049201c2ac61$454bdcc0$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> However, this all leads to a narrower search path for submodules. I want
> it to be *wider*.

Basically we want the same thing, but so either

1) __path__ manipulations are supported
2) or __path__ is deprecated and we get wide-importing by default (that means
the union of all relevant "directories" is considered and not just one)

indeed my use-case for 1) is to get 2).

Both solutions are OK, and yes the 2nd simplifies some things.

[ Then supporting path/to/a.zip/pkg1 in sys.path only would be still a feature
and would be easier to support or just Jython could want to support it. ]

My only remaining worry, once the above is sorted out, is whether the PEP302
scheme can be in the future expanded to support ihooks like functionalities
(like those used by Quixote) in a non-messy way.

regards.








From just@letterror.com  Wed Dec 25 22:48:59 2002
From: just@letterror.com (Just van Rossum)
Date: Wed, 25 Dec 2002 23:48:59 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
In-Reply-To: <5085E098-1854-11D7-BC72-000A27B19B96@oratrix.com>
Message-ID: <r01050400-1022-10D95D44185B11D7A7C4003065D5E7E4@[10.0.0.1]>

Jack Jansen wrote:

> Please don't wait too long, as I have no reasonable way to apply the
> patch on MacOS9, and I can imagine that MacPython-OS9 might encounter
> problems, with its own hooks in import.c. If you don't want to check
> it in on the trunk please do so on a branch.
> 
> Or did you test MacPython-OS9 interaction yourself?

No, but the current version of the patch doesn't do any significant path
munging, so I don't expect any problems. Nevertheless I'll send you the
relevant files (with Mac line endings) in a separate email.

Just


From just@letterror.com  Wed Dec 25 23:43:44 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 26 Dec 2002 00:43:44 +0100
Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31)
In-Reply-To: <049201c2ac61$454bdcc0$6d94fea9@newmexico>
Message-ID: <r01050400-1022-B880C586186211D7A7C4003065D5E7E4@[10.0.0.1]>

Samuele Pedroni wrote:

> Basically we want the same thing, but so either
> 
> 1) __path__ manipulations are supported 2) or __path__ is deprecated
> and we get wide-importing by default (that means the union of all
> relevant "directories" is considered and not just one)
> 
> indeed my use-case for 1) is to get 2).
> 
> Both solutions are OK, and yes the 2nd simplifies some things.

Hey, a consensus ;-)

Here's a concrete proposal:

  - deprecate pkg.__path__ *manipulation*, starting with Python 2.3a1.
  - pkg.__path__ will be set as before.
  - if len(pkg.__path__) > 1:
        issue DeprecationWarning
        search submodules on pkg.__path__, the old way.
    else:
        search submodules on sys.path, the new way.

(Fancier: keep a copy of __path__ in a separate variable, say
__orgpath__, issue a warning and invoke the old logic if __orgpath__ !=
__path__. This is robust even against alterations of __path__[0], but
seems a little overkill to me.)

This should be fully b/w compatible.

> [ Then supporting path/to/a.zip/pkg1 in sys.path only would be still
> a feature and would be easier to support or just Jython could want to
> support it. ]

Or: if Python ever grows a virtual file system, this feature could be
readded. I'd prefer to leave it out for now.

> My only remaining worry, once the above is sorted out, is whether the
> PEP302 scheme can be in the future expanded to support ihooks like
> functionalities (like those used by Quixote) in a non-messy way.

Right now there are two ways to go about addressing the issue:

A) turn/wrap the builtin Path importer into a real object on
sys.meta_path.

B) create a subset of ihooks.py as a new module (written in Python),
reimplementing some of the path import logic.

Either one should provide an API for adding file type handlers. (I'm not
so sure ihooks even provides a straightforward way for multiple
independent packages to each add their own file type handlers. For
example I don't see how Quixote could currently be compatible with other
packages doing similar things, short of chaining __import__ hooks. A new
solution should have it as a major goal to make this possible.)

B could be a prototype for A, so it seems logical to initially go for B.
This doesn't (IMO) need to be in first alpha, so there's more time to
experiment and discuss.

Just


From pedronis@bluewin.ch  Thu Dec 26 00:14:30 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Thu, 26 Dec 2002 01:14:30 +0100
Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31)
References: <r01050400-1022-B880C586186211D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <059601c2ac73$c329aca0$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> > Both solutions are OK, and yes the 2nd simplifies some things.
>
> Hey, a consensus ;-)

between the 2 of us, yes, let's wait what the others have to say.
Deprecating __path__ is maybe a small vs big change but is a subtle one.

> > My only remaining worry, once the above is sorted out, is whether the
> > PEP302 scheme can be in the future expanded to support ihooks like
> > functionalities (like those used by Quixote) in a non-messy way.
>
> Right now there are two ways to go about addressing the issue:
>
> A) turn/wrap the builtin Path importer into a real object on
> sys.meta_path.
>
> B) create a subset of ihooks.py as a new module (written in Python),
> reimplementing some of the path import logic.
>
> Either one should provide an API for adding file type handlers. (I'm not
> so sure ihooks even provides a straightforward way for multiple
> independent packages to each add their own file type handlers. For
> example I don't see how Quixote could currently be compatible with other
> packages doing similar things, short of chaining __import__ hooks. A new
> solution should have it as a major goal to make this possible.)
>
> B could be a prototype for A, so it seems logical to initially go for B.
> This doesn't (IMO) need to be in first alpha, so there's more time to
> experiment and discuss.

my view-point:

when you bring ihooks-type functionality in the picture, I see the following
separation of concerns:

- opaque importers:  fullname  -> module
- other importers:

* the importers allow to fish for file-like objects of some types
(corresponding to file extensions in filesystem-like cases)

* there is a way to register new types and way to convert file-like objects
into code/modules.

instead of asking directly for a module, one would ask the importer
for <fullname>[.__init__] and  e.g. types  (module,code,source (py), bytecode
(pyc), plt, pltc)

[module and code are special types ]

the importers would return some proxies for all relevant objects, the proxies
would allow to possibly "open" and "read" the object and will present an
optional timestamp.

The converters would be used to make code/modules out of the objects.

regards.















From guido@python.org  Thu Dec 26 01:15:00 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 25 Dec 2002 20:15:00 -0500
Subject: Packages & __path__ (was:Re: [Python-Dev] Python 2.3a1 release -- Dec 31)
In-Reply-To: Your message of "Thu, 26 Dec 2002 01:14:30 +0100."
 <059601c2ac73$c329aca0$6d94fea9@newmexico>
References: <r01050400-1022-B880C586186211D7A7C4003065D5E7E4@[10.0.0.1]>
 <059601c2ac73$c329aca0$6d94fea9@newmexico>
Message-ID: <200212260115.gBQ1F0207353@pcp02138704pcs.reston01.va.comcast.net>

> Deprecating __path__ is maybe a small vs big change but is a subtle one.

Especially when we just had a discussion in Zope3-dev where the
conclusion was that we wanted to extend __path__ to solve a certain
problem with separate distribution of self-contained subpackages of
the zope package.

I'll read the whole thread tomorrow (when I'm back to work) and will
comment more then.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From bckfnn@worldonline.dk  Thu Dec 26 10:19:45 2002
From: bckfnn@worldonline.dk (Finn Bock)
Date: Thu, 26 Dec 2002 11:19:45 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
References: <r01050400-1022-C8E9667517E711D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <3E0AD7C1.4010400@worldonline.dk>

[Samuele]

>>further this change is backward incompatible with what is allowed by
>>Jython 2.1,
>>
>>it was considered a feature to be able to put
>>
>>path/to/a.zip/python
>>
>>in sys.path, so that the a.b package would be looked up under
>>
>>path/to/a.zip/python/a/b.

[Just]

> PEP 273 doesn't document it as such; it only says it's needed for
> package imports. Also, to be honest, my implementation had some issues
> with that usage: it would look for the plain .zip archive in
> sys.path_importer_cache, which would obviously not be found, causing the
> zip file index to be read again for every package directory. The only
> solution I thought of that could solve that is for the zipimporter
> object to *add* entries to sys.path_importer_cache itself, and I found
> it bad enough already that it _read_ from the cache itself in the
> previous version. It's a messy feature :-(.
> 
> I personally don't care about this feature; it's easy enough to package
> the archive so that it's not needed.

That may be true for CPython but it isn't for jython where the toplevel 
namespace in a zip file is taken already for java classes. By far the 
most natural way to package a jython/java application would be to put 
the jython modules in the same .zip (or .jar) file as the rest of the 
java application and that is best achieved by putting the python modules 
under a seperate directory in the zip file.

So Jython cares about this feature. So mush that we may have to be 
incompatible on this point with CPython.

regards,
finn



From just@letterror.com  Thu Dec 26 12:06:10 2002
From: just@letterror.com (Just van Rossum)
Date: Thu, 26 Dec 2002 13:06:10 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
In-Reply-To: <3E0AD7C1.4010400@worldonline.dk>
Message-ID: <r01050400-1022-7534656918CA11D7A7C4003065D5E7E4@[10.0.0.1]>

Finn Bock wrote:

> That may be true for CPython but it isn't for jython where the
> toplevel namespace in a zip file is taken already for java classes.
> By far the most natural way to package a jython/java application
> would be to put the jython modules in the same .zip (or .jar) file as
> the rest of the java application and that is best achieved by putting
> the python modules under a seperate directory in the zip file.
> 
> So Jython cares about this feature. So mush that we may have to be
> incompatible on this point with CPython.

I wonder, though, how big a deal is this incompatibility? Can Jython
load modules from .pyc files in zip files? Can CPython import modules
from .jar files? In other words, can zip/jar files ever be shared
between Jython and CPython?

Just


From pedronis@bluewin.ch  Thu Dec 26 12:10:33 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Thu, 26 Dec 2002 13:10:33 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
References: <r01050400-1022-7534656918CA11D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <002301c2acd7$cb1b3820$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
> I wonder, though, how big a deal is this incompatibility? Can Jython
> load modules from .pyc files in zip files? Can CPython import modules
> from .jar files? In other words, can zip/jar files ever be shared
> between Jython and CPython?

the real problem is that in order to support this, if Jython/Python keep
__path__, Jython has to put different values in there wrt
to Python.

If __path__ goes deprecated then it's just a sys.path contents issue, so what
Jython does could be considered just a small extension.

regards.



From jim@interet.com  Fri Dec 27 15:19:59 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 27 Dec 2002 10:19:59 -0500
Subject: [Python-Dev] New Import Hooks PEP, a first draft (and req. for
 PEP #)
References: <r01050400-1022-6B419BC4144811D7A7C4003065D5E7E4@[10.0.0.1]> <znr0zf2y.fsf@python.net> <3E037BFE.7020509@interet.com> <n2m-g.isxow9kl.fsf@morpheus.demon.co.uk>
Message-ID: <3E0C6F9F.4060407@interet.com>

Paul Moore wrote:

> It's an interesting idea, but like others, I think that it's not going
> to work right in practice. The main problem is that you don't explain
> how to implement this version of imp.find_module in terms of the
> importer protocol defined in the hooks PEP.

I am exploring the idea of adding the feature to imp.find/load_module
keeping the existing logic.  I am not using the hooks PEP because I
think the hooks PEP is a bad idea.

Anyway, after some thought, I think my idea of a file-like object
is not good.  The easiest solution is to add a get_data() to
go with find_module() and load_module().  The function signature
of get_data() (or maybe "get_module()") is the same as that
of load_module(), and the interface is:

     file, pathname, desc = imp.find_module(name, path=p, suffix=s)
     bytes = imp.get_data(name, file, pathname, desc)

JimA



From guido@python.org  Fri Dec 27 17:59:52 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 27 Dec 2002 12:59:52 -0500
Subject: [Python-Dev] PEP 302 and __path__
In-Reply-To: Your message of "Thu, 26 Dec 2002 13:10:33 +0100."
 <002301c2acd7$cb1b3820$6d94fea9@newmexico>
References: <r01050400-1022-7534656918CA11D7A7C4003065D5E7E4@[10.0.0.1]>
 <002301c2acd7$cb1b3820$6d94fea9@newmexico>
Message-ID: <200212271759.gBRHxqx21233@pcp02138704pcs.reston01.va.comcast.net>

I've thought a lot about __path__, and my conclusion is that I want to
keep it.  It was put in as a feature of packages long ago (surviving a
major redesign, when other features were dropped) and IMO serves an
important purpose.  For example, some 3rd party packages (e.g. PMW)
use __path__ to select the most recently installed version when the
package is imported.  I don't like the idea of widening the package
search path by default -- I can see all sorts of confusion when two
versions of a package are on the path, and I prefer the (default)
situation where the first one found hides the second one completely.
That said, there are cases where a widened search path is desirable,
and this can be achieved by explicit __path__ manipulation; that's why
I recently added pkgutil.py (there's a potential use of this in
Zope3).

I also like the idea of allowing references to directories *inside* a
zip archive (e.g. /path/to/foo.zip/dir/subdir/) on sys.path and on
__path__.  This seems pretty natural, and Jython already uses this.
Using a smidgeon of caching, it should be pretty efficient to find the
zip file in the path (even if it doesn't have a .zip extension; I
don't think we should require that).  Of course, there is an issue
when the OS's path separator isn't the same as the zipfile's path
separator; I think we should use the OS's path separator throughout
and translate that to the zipfile's path separator (always "/" I
believe?) internally.  We should also silently strip a leading "/" on
the paths used in the zipfile (i.e., it shouldn't matter if the
zipfile index uses /foo/bar.py or foo/bar.py -- in both cases you
could refer to this as /path/to/foo.zip/foo/bar.py.  For directories,
a trailing slash should also be optional (for files it should be
illegal).

Then __file__ for a package or module loaded from a zipfile should be
set to the a path as above (e.g. /path/to/foo.zip/foo/bar.py) and
__path__ for a package loaded from a zipfile should be initialized to
e.g.  ['/path/to/foo.zip/foo'].

I'm not requiring *all* importers to follow this convention.  It's
fine if e.g. the "freeze" importer does something else (although I
wouldn't be surprised if "freeze" ends up being deprecated once we
have zip import as a standard feature).  But for importers that map to
something reasonably close to a (read-only) hierarchical file system,
it seems useful to use OS-filename-like syntax in __file__ and
__path__.

Now, for such importers, it would be nice if we could use such paths
to extract other data from the importer as well.  I think that the
right API for this would be some function living in the imp module:
you pass it a path and it returns the data as a string, or raises an
IOError (or subclass thereof) instance if it can't find the data.
Let's call this API imp.get_data(filename).  We'll see how it
interfaces to importer/loader objects in a minute.

I also would like to propose a new API to find modules:
imp.get_loader(name[, path]).  This should return a module loader
object, or None if the module isn't found; it should raise an
exception only if something unexpected went wrong.  Once we have a
module loader object, loader.load_module(name[, path]) should load
(and return) the requested module.  The name argument in both cases
should be the fully dotted module name; the path argument should be
omitted or None to load a toplevel module or package (no dot in the
name), and it should be the package path to load a submodule or
subpackage.

Note that the package path can contain multiple entries.
imp.get_loader() will have to probe each in turn until it finds one
that has the requested module, and then return the corresponding
loader.  That loader, when its load_module() method is called, may use
or ignore the path passed in.

I don't want to add a separate meta-path to a package; it seems
overkill, especially since we don't even have a good use case for
meta-path in the normal case.

I'm trying to write up pseudo-code to describe the whole setup more
precisely, but it's taking more time than expected, so I'll send this
mail with my intentions out first.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gward@python.net  Fri Dec 27 21:05:34 2002
From: gward@python.net (Greg Ward)
Date: Fri, 27 Dec 2002 16:05:34 -0500
Subject: [Python-Dev] smtplib docs: has_option() or has_extn()?
Message-ID: <20021227210534.GA5107@cthulhu.gerg.ca>

Doc/lib/libsmtplib.tex refers to a mythical has_option() method of SMTP
objects.  (The string 'has_option' occurs nowhere in Lib/smtplib.py.
Neither does 'hasoption'.)  I'm pretty sure that it should be talking
about has_extn(), ie.:

--- lib/libsmtplib.tex  24 Mar 2002 15:41:40 -0000      1.22
+++ lib/libsmtplib.tex  27 Dec 2002 21:03:40 -0000
@@ -138,9 +138,9 @@
 Identify yourself to an ESMTP server using \samp{EHLO}.  The hostname
 argument defaults to the fully qualified domain name of the local
 host.  Examine the response for ESMTP option and store them for use by
-\method{has_option()}.
+\method{has_extn()}.
 
-Unless you wish to use \method{has_option()} before sending
+Unless you wish to use \method{has_extn()} before sending
 mail, it should not be necessary to call this method explicitly.  It
 will be implicitly called by \method{sendmail()} when necessary.
 \end{methoddesc}

This look wrong to anyone?  (Or is my CVS out-of-date?)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
"Very funny, Scotty.  Now beam my *clothes* down."


From jim@interet.com  Fri Dec 27 20:58:12 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 27 Dec 2002 15:58:12 -0500
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
References: <200212241930.gBOJUYD10795@odiug.zope.com>
Message-ID: <3E0CBEE4.1020902@interet.com>

Let's focus on PEP 302 as a new import hook mechanism, not
as a way to implement zip imports.  Just's PEP 302 code can
be used to implement zip imports without making his import
hooks part of the sys module and thus making them a public
feature of Python forever.  THIS IS NOT ABOUT ZIP IMPORTS!

Python already has an import hook, namely __import__.  PEP
302 adds three more hooks: sys.path_hooks, sys.meta_path and
sys.path_importer_hooks.  The idea of four import hooks is
already fishy.

PEP 302 enables non-strings on sys.path, and adds two new
Python objects "importer" and "loader".  It changes the meaning
of imp.find_module() and imp.load_module(), and adds a new
imp.find_module2().  It changes the meaning of __import__.
It proposes to deprecate __path__ manipulations.

That is a lot of external changes.  That is a lot of code
written in C.

I think the proper import hook design is to write Python's
import mechanism in Python along the lines of Greg's imputil.py
and Gordon's iu.py.  Import.c would be responsible for flat
non-package imports from directories and zip files, including
the import of the real importer iu.py.  The imp module would be
extended with simple C import utilities that can be used to
speed up iu.  Once iu.py is imported (probably from site),
all imports are satisfied using the iu module.

To provide custom import hooks, the user overrides the iu
module by writing Python code.  For example, site.py can
attempt an import of custom_iu before importing iu, and the
user provides custom_iu.py, probably by copying iu.py.
I am not sure of the best way to do the override, but I am
sure it is done in Python code. That enables the user to
create custom hooks in Python, while relying on the source
iu.py and the utilities in the imp module for basic
functionality.

If sys.path_hooks and the other hooks described in PEP 302
are approved, they can be implemented in iu.py in Python.

This design still requires C support for zip imports, but
there are two implementations for that available (from JimA
and Just).  Other problems such as bootstrapping are easily
solved.

I am tired of these endless import hook discussions, which
always seem to start from nifty ideas instead of from a
formal solicitation of hook requirements.  I don't object to
useful features, but I don't think anything other than easy
replacement of the Python import mechanism will bring this
perennial topic to an end.

JimA



From guido@python.org  Fri Dec 27 22:15:23 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 27 Dec 2002 17:15:23 -0500
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: Your message of "Fri, 27 Dec 2002 15:58:12 EST."
 <3E0CBEE4.1020902@interet.com>
References: <200212241930.gBOJUYD10795@odiug.zope.com>
 <3E0CBEE4.1020902@interet.com>
Message-ID: <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net>

> Let's focus on PEP 302 as a new import hook mechanism, not
> as a way to implement zip imports.  Just's PEP 302 code can
> be used to implement zip imports without making his import
> hooks part of the sys module and thus making them a public
> feature of Python forever.  THIS IS NOT ABOUT ZIP IMPORTS!

Agreed.  It's about making it easier to do things *like* zip import.

> Python already has an import hook, namely __import__.  PEP
> 302 adds three more hooks: sys.path_hooks, sys.meta_path and
> sys.path_importer_hooks.  The idea of four import hooks is
> already fishy.

There's no path_importer_hooks; there's path_importer_cache, but
that's not a new hook, it's a cache for the path_hooks hook.  So I
think the PEP proposes only two new hooks.

Two new hooks is arguably still too much, but the existing __import__
hook is inadequate because it requires you to reimplement too much
functionality (the PEP argues this point convincingly IMO).  So we
need at least one new hook.  Personally, I think sys.meta_path is the
lesser important of the two hooks proposed by the PEP.  It would be
needed if you have to override the standard builtin or frozen import
behavior, but you can already do that with the heavier gun of
overriding __import__.

Other than that, you can hook up arbitrary weird importers by placing
magic-cookie strings in sys.path (e.g. of the form "<blah>") and
registering a path hook that looks for the magic cookie.  (Long ago I
had an idea to do this for builtin and frozen imports too, so you
could control the relative priority of builtin/frozen modules relative
to some directories.  But I never found a use case.  If someone wants
this, though, it's easily added to the path_hooks feature.)

Magic cookie strings are backwards compatible: from inspecting code
that does something with sys.path, it looks like there are many places
that assume sys.path contains only strings, but almost none that
assume that all those strings are valid directory names.  Typical code
uses a sys.path item as input to os.path.isdir() or os.path.join() --
these require strings but don't make other assumptions.  In the end
there's usually something that passes the result to open() or
os.stat() to see if it exists.  Magic cookies will cause this test to
fail, but the code is prepared for such failure -- however it's not
prepared for TypeError coming out of a string concatenation.

> PEP 302 enables non-strings on sys.path, and adds two new
> Python objects "importer" and "loader".  It changes the meaning
> of imp.find_module() and imp.load_module(), and adds a new
> imp.find_module2().  It changes the meaning of __import__.
> It proposes to deprecate __path__ manipulations.

Yes, this is definitely too much.  I'd like to limit this to
implementing sys.path_hooks -- there should be only one way to do it.
We might still want to add one new API to imp, to access the new
module-finding functionality (find_module2 is a poor choice of name
though).

> That is a lot of external changes.  That is a lot of code
> written in C.

Um, last I looked, most of the code written in C was specific to the
zip importer; only a relatively small amount of code was added to
import.c (about 10%).  If we get rid of the meta_path hook, it will be
less; if we drop non-string objects on sys.path, less again.

> I think the proper import hook design is to write Python's
> import mechanism in Python along the lines of Greg's imputil.py
> and Gordon's iu.py.  Import.c would be responsible for flat
> non-package imports from directories and zip files, including
> the import of the real importer iu.py.  The imp module would be
> extended with simple C import utilities that can be used to
> speed up iu.  Once iu.py is imported (probably from site),
> all imports are satisfied using the iu module.

That's a design that I have had in mind long ago, but I don't see it
happening soon, because it would be a much larger overhaul of
import.c.  Also, there are more risks: if import.c somehow can't find
iu.py, it's hosed; and I fear that it could be a significant
performance risk in its first implementation (I vaguely remember that
Greg did some timing tests with imputil.py that confirmed this).

> To provide custom import hooks, the user overrides the iu
> module by writing Python code.  For example, site.py can
> attempt an import of custom_iu before importing iu, and the
> user provides custom_iu.py, probably by copying iu.py.
> I am not sure of the best way to do the override, but I am
> sure it is done in Python code. That enables the user to
> create custom hooks in Python, while relying on the source
> iu.py and the utilities in the imp module for basic
> functionality.

More machinery that's not yet designed and implemented.

I'd like to get something implemented by Monday, that supports zip
import and *some* hookability.  A scaled-down version of Just's code
seems the only realistic possibility.

> If sys.path_hooks and the other hooks described in PEP 302
> are approved, they can be implemented in iu.py in Python.
> 
> This design still requires C support for zip imports, but
> there are two implementations for that available (from JimA
> and Just).  Other problems such as bootstrapping are easily
> solved.
> 
> I am tired of these endless import hook discussions, which
> always seem to start from nifty ideas instead of from a
> formal solicitation of hook requirements.  I don't object to
> useful features, but I don't think anything other than easy
> replacement of the Python import mechanism will bring this
> perennial topic to an end.

I'm not so sure.  In practice, hooks are used for two things: import
from other media than directories (e.g. zip files), and supporting
additional filename extensions that trigger special transformations.
But the latter need is much less common than the former (the only real
example I know of is Quixote) and it's pretty much orthogonal to it.
Neither should require you to replace __import__, as they currently
do.

(I've got to run now -- more over the weekend as my family gives me
some time off. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Fri Dec 27 21:27:23 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Fri, 27 Dec 2002 16:27:23 -0500
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
References: <r01050400-1022-C8E9667517E711D7A7C4003065D5E7E4@[10.0.0.1]>
Message-ID: <3E0CC5BB.8030200@interet.com>

Just van Rossum wrote:
> From: "Samuele Pedroni" <pedronis@bluewin.ch>
>>it was considered a feature to be able to put
>>path/to/a.zip/python
>>in sys.path, so that the a.b package would be looked up under
>>path/to/a.zip/python/a/b.
> 
> PEP 273 doesn't document it as such; it only says it's needed for
> package imports.

The cornerstone of PEP 273 is Directory Equivalence, a zipped
directory must act like the directory.  And my implementation
does provide this feature.

> Also, to be honest, my implementation had some issues
> with that usage: it would look for the plain .zip archive in
> sys.path_importer_cache, which would obviously not be found, causing the
> zip file index to be read again for every package directory.

Mine reads the zip directory only once, and caches the whole
directory tree.  The payoff from the heavy mods to import.c, I guess.

JimA



From just@letterror.com  Fri Dec 27 22:33:18 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 27 Dec 2002 23:33:18 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <3E0CBEE4.1020902@interet.com>
Message-ID: <r01050400-1022-3C57A01E19EB11D7A7C4003065D5E7E4@[10.0.0.2]>

James C. Ahlstrom wrote:

> Let's focus on PEP 302 as a new import hook mechanism, not
> as a way to implement zip imports.  Just's PEP 302 code can
> be used to implement zip imports without making his import
> hooks part of the sys module and thus making them a public
> feature of Python forever.  THIS IS NOT ABOUT ZIP IMPORTS!

Indeed it isn't.

> Python already has an import hook, namely __import__. 

Yes, and the PEP explains clearly what the problems with __import__ (as
a hook) are.

> PEP 302 adds three more hooks: sys.path_hooks, sys.meta_path and
> sys.path_importer_hooks. 

The latter does not exist. There are *two* new hooks: sys.meta_path and
sys.path_hooks. Then there is sys.path_importer_cache, which caches the
results of sys.path_hooks.

> The idea of four import hooks is already fishy.

Well, __import__ is fishy, the new hooks

> PEP 302 enables non-strings on sys.path, and adds two new
> Python objects "importer" and "loader".  It changes the meaning
> of imp.find_module() and imp.load_module(), and adds a new
> imp.find_module2(). 

Yes. The latter remains to be seen (it's not even implemented yet), and
I see Guido just proposed an alternative. I haven't had time yet to read
his full post.

> It changes the meaning of __import__.

Where did you get that idea?

> It proposes to deprecate __path__ manipulations.

It certainly does not. *I* proposed that, but not in PEP 302. Guido is
against it, so that settles that.

> That is a lot of external changes.  That is a lot of code
> written in C.

Except it's not a lot of code.

> I think the proper import hook design is to write Python's
> import mechanism in Python along the lines of Greg's imputil.py
> and Gordon's iu.py. 

This is what iu.py *is*. But iu.py is more: while being a close
reimplementation of all the semantic details, it's a better abstraction
of the Python import mechanism than import.c is. PEP 302 attempts to
expose the key benefits of this better abstraction to Python.

> Import.c would be responsible for flat
> non-package imports from directories and zip files, including
> the import of the real importer iu.py.  The imp module would be
> extended with simple C import utilities that can be used to
> speed up iu.  Once iu.py is imported (probably from site),
> all imports are satisfied using the iu module.

You can do that now by simply using iu.py.

> To provide custom import hooks, the user overrides the iu
> module by writing Python code.  For example, site.py can
> attempt an import of custom_iu before importing iu, and the
> user provides custom_iu.py, probably by copying iu.py.
> I am not sure of the best way to do the override, but I am
> sure it is done in Python code. That enables the user to
> create custom hooks in Python, while relying on the source
> iu.py and the utilities in the imp module for basic
> functionality.
> 
> If sys.path_hooks and the other hooks described in PEP 302
> are approved, they can be implemented in iu.py in Python.

This is backwards: iu.py implements a superset of PEP 302 (with
different details). So you have that now.

> This design still requires C support for zip imports, but
> there are two implementations for that available (from JimA
> and Just).  Other problems such as bootstrapping are easily
> solved.
> 
> I am tired of these endless import hook discussions, which
> always seem to start from nifty ideas instead of from a
> formal solicitation of hook requirements. 

PEP 302 started from the real requirements of a real use case:
zipimport. Added to that are ideas taken from an extremely well though
out model that was developed to solve real problems (iu.py).

> I don't object to
> useful features, but I don't think anything other than easy
> replacement of the Python import mechanism will bring this
> perennial topic to an end.

This is no improvement. You can replace the import mechanism by
overriding __import__. This sucks for most purposes as the import
mechanism is very complicated and contains many subtle details and
pitfalls. PEP 302 allows customizing *parts* of the import mechanism
without having to deal with most of these pitfalls and complications. It
allows completely independend components to add hooks that will work
together seamlessy. This is not true for replacing __import__.

Just


From just@letterror.com  Fri Dec 27 22:37:50 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 27 Dec 2002 23:37:50 +0100
Subject: [Python-Dev] Python 2.3a1 release -- Dec 31
In-Reply-To: <3E0CC5BB.8030200@interet.com>
Message-ID: <r01050400-1022-D7F417B119EB11D7A7C4003065D5E7E4@[10.0.0.2]>

James C. Ahlstrom wrote:

> Just van Rossum wrote:
> > From: "Samuele Pedroni" <pedronis@bluewin.ch>
> >>it was considered a feature to be able to put path/to/a.zip/python
> >>in sys.path, so that the a.b package would be looked up under
> >>path/to/a.zip/python/a/b.
> > 
> > PEP 273 doesn't document it as such; it only says it's needed for
> > package imports.
> 
> The cornerstone of PEP 273 is Directory Equivalence, a zipped
> directory must act like the directory.  And my implementation does
> provide this feature.
> 
> > Also, to be honest, my implementation had some issues with that
> > usage: it would look for the plain .zip archive in
> > sys.path_importer_cache, which would obviously not be found,
> > causing the zip file index to be read again for every package
> > directory.
> 
> Mine reads the zip directory only once, and caches the whole
> directory tree.  The payoff from the heavy mods to import.c, I guess.

I had a long talk with Guido yesterday, and I'm going to go back to
supporting the archive.zip/subdir/ feature as well as adding a proper
archive.zip/subdir/ string to __path__. I hope to do this before monday,
if my parents-in-law allow me ;-)

Just


From just@letterror.com  Fri Dec 27 22:59:48 2002
From: just@letterror.com (Just van Rossum)
Date: Fri, 27 Dec 2002 23:59:48 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-F711A50A19EE11D7A7C4003065D5E7E4@[10.0.0.2]>

Guido van Rossum wrote:

> Two new hooks is arguably still too much, but the existing __import__
> hook is inadequate because it requires you to reimplement too much
> functionality (the PEP argues this point convincingly IMO).  So we
> need at least one new hook.  Personally, I think sys.meta_path is the
> lesser important of the two hooks proposed by the PEP.  It would be
> needed if you have to override the standard builtin or frozen import
> behavior, but you can already do that with the heavier gun of
> overriding __import__.

The sys.meta_path feature adds 32 lines to import.c, it's *really*
useful and easy to use. I would be *extremely* sad to see it go.

(I had an immediate use for it in test_importhooks.py: after a test is
run I wanted to unload the modules that were imported during the test.
The ImportTracker is a meta importer and only *records* imports. It's
seven lines long and helped me clean up the test script quite a bit. Not
a typical use case perhaps, but to me still demonstrates the power of
meta_path quite well. Doing the same with __import__ is of course
possible, but is much more cumbersome and doesn't have the same
semantics.)

> > PEP 302 enables non-strings on sys.path, and adds two new
> > Python objects "importer" and "loader".  It changes the meaning
> > of imp.find_module() and imp.load_module(), and adds a new
> > imp.find_module2().  It changes the meaning of __import__.
> > It proposes to deprecate __path__ manipulations.
> 
> Yes, this is definitely too much. 

Removing the importer-object-on-sys.path feature I could live with. It's
clearly not needed, and for quick experimentation sys.meta_path is your
pal (another reason I don't want to see it go). It's only there because
it was easy: it's 17 lines in import.c.

> I'd like to limit this to
> implementing sys.path_hooks -- there should be only one way to do it.

There should be only one way to do it if import.c did it only one way.
Overriding or emulating builtin and frozen imports *must* be possible,
and sys.meta_path is the feature for it. A key aspect of sys.meta_path
is that it allows to refactor import.c in a really nice way; in the
future, sys.meta_path would look like this:

    [BuiltinImporter, FrozenImporter, PathImporter]

sys.meta_path is then the central import dispatcher. PathImporter would
be responsible for sys.path/pkg.__path__ handling and invoking
sys.path_hooks and dealing with sys.path_import_cache. It is *key* to
PEP 302, and to me simply a killer feature.

Just


From lalo@laranja.org  Fri Dec 27 23:03:41 2002
From: lalo@laranja.org (Lalo Martins)
Date: Fri, 27 Dec 2002 21:03:41 -0200
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <r01050400-1022-F711A50A19EE11D7A7C4003065D5E7E4@[10.0.0.2]>
References: <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net> <r01050400-1022-F711A50A19EE11D7A7C4003065D5E7E4@[10.0.0.2]>
Message-ID: <20021227230341.GA15340@laranja.org>

On Fri, Dec 27, 2002 at 11:59:48PM +0100, Just van Rossum wrote:
> 
> The sys.meta_path feature adds 32 lines to import.c, it's *really*
> useful and easy to use. I would be *extremely* sad to see it go.
> 
> (I had an immediate use for it in test_importhooks.py: after a test is
> run I wanted to unload the modules that were imported during the test.
> The ImportTracker is a meta importer and only *records* imports. It's
> seven lines long and helped me clean up the test script quite a bit. Not
> a typical use case perhaps, but to me still demonstrates the power of
> meta_path quite well. Doing the same with __import__ is of course
> possible, but is much more cumbersome and doesn't have the same
> semantics.)

Just my 1.5c - PyUnit has a similar use case.  It is documented (well) at
http://pyunit.sourceforge.net/notes/reloading.html

[]s,
                                               |alo
                                               +----
--
            Those who trade freedom for security
               lose both and deserve neither.
--
http://www.laranja.org/                mailto:lalo@laranja.org
         pgp key: http://www.laranja.org/pessoal/pgp

Eu jogo RPG! (I play RPG)         http://www.eujogorpg.com.br/
GNU: never give up freedom                 http://www.gnu.org/


From macquigg@cadence.com  Sat Dec 28 00:41:33 2002
From: macquigg@cadence.com (David MacQuigg)
Date: Fri, 27 Dec 2002 16:41:33 -0800
Subject: [Python-Dev] linktree.py
Message-ID: <9A77785A255522459FDD2D24C94B27D19DE7B8@exmbx02sj.cadence.com>

Hello,

I'm new to Python, and I'm very impressed with the potential of this =
language to simplify much of the software my company provides.  As a =
first project, I'm looking to replace some tricky scripts written in csh =
for setting up our tool hierarchies.  An example is a script to =
construct a "mock hierarchy" out of some patch files and links to a =
production hierarchy.  Our hierarchies are typically two gigabytes, =
thousands of files, lots of strange links, many of which are broken.  So =
the script needs to be fairly robust.  The functions in 'shutil' won't =
do it.  'copytree' works, but the cost is a huge amount of disk space.  =
A mock hierarchy takes not much more space than the patch files =
themselves.

After a brief search on the Python website, I couldn't find a suitable =
utility, and decided to write my own (see below).  It seems like this =
could be a useful, general-purpose addition to 'shutil'.

I've tested this on a large hierarchy on a Solaris workstation, and it =
seems to work.  There are some dark corners which worry me however, so I =
thought I would get some advice from the experts.  Has anything like =
this already been done or is it planned?  Would a function like this be =
a useful addition to 'shutil'?  I'm intending to continue work on it, =
and will be glad to contribute it to the library.

import os, sys
from os.path import *

def linktree(oldtree, newtree):
  """linktree(oldtree, newtree) -> None
  Leave 'oldtree' undisturbed.  Recursively add links to 'newtree', =
until it
  looks just like 'oldtree', but with the new files patched in.=20
  """
  names =3D os.listdir(oldtree)
  print 'Candidates: ', names
  for name in names:
    oldpath =3D abspath(join(oldtree, name))
    newpath =3D abspath(join(newtree, name))
    if not exists(newpath) and not islink(newpath):
      if not exists(oldpath):  # Don't link to a bad link.
        print '*** Error ***\nPath does not exist:\n', oldpath
        continue
      print 'New link: ', newpath
      os.symlink(oldpath, newpath)  # OK if oldpath is a good link.
    elif isfile(newpath) and not islink(newpath):
      print 'As is: ', newpath=20
      pass  # Leave new files as is.
    elif isdir(newpath) and not islink(newpath):
      print 'Down one level \n', newpath
      linktree(oldpath, newpath)  # Recursive call
      print 'Up one level'
    else:
      print '*** Error ***\nNot a file or directory:\n', newpath

- Dave

*************************************************************     *
* David MacQuigg               * email:  macquigg@cadence.com  *  *
* Principal Product Engineer   * phone:  USA 520-721-4583      *  *  *
* Analog Artist                                                *  *  *
*                                * 9320 East Mikelyn Lane       * * *
* Cadence Design Systems, Inc.   * Tucson, Arizona 85710          *
*************************************************************     *=20


From guido@python.org  Sat Dec 28 01:09:09 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 27 Dec 2002 20:09:09 -0500
Subject: [Python-Dev] linktree.py
In-Reply-To: Your message of "Fri, 27 Dec 2002 16:41:33 PST."
 <9A77785A255522459FDD2D24C94B27D19DE7B8@exmbx02sj.cadence.com>
References: <9A77785A255522459FDD2D24C94B27D19DE7B8@exmbx02sj.cadence.com>
Message-ID: <200212280109.gBS199508938@pcp02138704pcs.reston01.va.comcast.net>

Sorry, you've reached the python *developers* list.  For review of
your code, you're better off in comp.lang.python.  If you want to
contribute a patch, please check here:
http://www.python.org/dev/devfaq.html#patches

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Sat Dec 28 10:24:32 2002
From: just@letterror.com (Just van Rossum)
Date: Sat, 28 Dec 2002 11:24:32 +0100
Subject: [Python-Dev] PEP 302 and __path__
In-Reply-To: <200212271759.gBRHxqx21233@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-9072F5D01A4E11D7A7C4003065D5E7E4@[10.0.0.2]>

Guido van Rossum wrote:

> I also would like to propose a new API to find modules:
> imp.get_loader(name[, path]). 

Hey, that's exactly what I proposed, with a better name ;-)

> This should return a module loader
> object, or None if the module isn't found; it should raise an
> exception only if something unexpected went wrong.  Once we have a
> module loader object, loader.load_module(name[, path]) should load
> (and return) the requested module. 

loader.load_module() doesn't need the optional path argument and is
indeed spec'ed without it in the PEP. This is symmetrical with
imp.load_module().

Just


From pedronis@bluewin.ch  Sat Dec 28 12:57:31 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Sat, 28 Dec 2002 13:57:31 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
References: <r01050400-1022-F711A50A19EE11D7A7C4003065D5E7E4@[10.0.0.2]>
Message-ID: <003d01c2ae70$af669120$6d94fea9@newmexico>

From: "Just van Rossum" <just@letterror.com>
>
> The sys.meta_path feature adds 32 lines to import.c, it's *really*
> useful and easy to use. I would be *extremely* sad to see it go.
>
> (I had an immediate use for it in test_importhooks.py: after a test is
> run I wanted to unload the modules that were imported during the test.
> The ImportTracker is a meta importer and only *records* imports.  ...

I agree with Guido that special cookies can be used for normal importers.

I see also the point for such really _meta_ importers. But if one has many of
them how the code that installs them respectively the user decide on the order
of invocation.

It seems that theoretically to let code programatically alter meta_path,
there's a need for categories/priorities for meta importers:

- loggging importers come before everything
- global importers after
etc

Lacking this, it seems equivalent to installing __import__ hooks apart a tiny
bit of convenience.

> There should be only one way to do it if import.c did it only one way.
> Overriding or emulating builtin and frozen imports *must* be possible,
> and sys.meta_path is the feature for it. A key aspect of sys.meta_path
> is that it allows to refactor import.c in a really nice way; in the
> future, sys.meta_path would look like this:
>
>     [BuiltinImporter, FrozenImporter, PathImporter]
>
> sys.meta_path is then the central import dispatcher. PathImporter would
> be responsible for sys.path/pkg.__path__ handling and invoking
> sys.path_hooks and dealing with sys.path_import_cache. It is *key* to
> PEP 302, and to me simply a killer feature.

This would be nice, but if the above point is not resolved and this is not
concretely implemented, we risk to have meta_path used in non-intended ways or
be as much problematic as __import__.

Lacking precise policies and clear-cut commitment for its future uses, it's
better left out (it can be added in the future).








From pedronis@bluewin.ch  Sat Dec 28 13:31:14 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Sat, 28 Dec 2002 14:31:14 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
References: <r01050400-1022-F711A50A19EE11D7A7C4003065D5E7E4@[10.0.0.2]> <003d01c2ae70$af669120$6d94fea9@newmexico>
Message-ID: <000d01c2ae75$65030aa0$6d94fea9@newmexico>

>
> Lacking this, it seems equivalent to installing __import__ hooks apart a tiny
> bit of convenience.
>

ok, not irrelevant detail, with meta_path hooks can be removed, still where to
put a hook in meta_path is problematic.



From just@letterror.com  Sat Dec 28 13:37:34 2002
From: just@letterror.com (Just van Rossum)
Date: Sat, 28 Dec 2002 14:37:34 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <003d01c2ae70$af669120$6d94fea9@newmexico>
Message-ID: <r01050400-1022-8A8BBD231A6911D7A7C4003065D5E7E4@[10.0.0.2]>

Samuele Pedroni wrote:

> I agree with Guido that special cookies can be used for normal
> importers.

I find this rather ugly, and would only do that if it's crucial that the
pathless importer should be invoked somewhere in the middle of sys.path.
And I don't have a use case for that. There are plenty of use cases for
sys.meta_path.

> I see also the point for such really _meta_ importers. But if one has
> many of them how the code that installs them respectively the user
> decide on the order of invocation.

I have no idea what you mean by this.

> It seems that theoretically to let code programatically alter
> meta_path, there's a need for categories/priorities for meta
> importers:
> 
> - loggging importers come before everything - global importers after
> etc
> 
> Lacking this, it seems equivalent to installing __import__ hooks
> apart a tiny bit of convenience.

I disagree that it's only a "tiny bit" of convenience:

- PEP-302-style importers don't have to deal with the "is this module
already loaded" issue, it will only be invoked for new imports.

- They don't have to deal with complete dotted name imports, they only
get invoked for the head of a dotted name. The rules for dotted name
imports are very subtle and are all taken care of by the surrounding
code, so this really is a big win.

> > There should be only one way to do it if import.c did it only one
> > way. Overriding or emulating builtin and frozen imports *must* be
> > possible, and sys.meta_path is the feature for it. A key aspect of
> > sys.meta_path is that it allows to refactor import.c in a really
> > nice way; in the future, sys.meta_path would look like this:
> >
> > [BuiltinImporter, FrozenImporter, PathImporter]
> >
> > sys.meta_path is then the central import dispatcher. PathImporter
> > would be responsible for sys.path/pkg.__path__ handling and
> > invoking sys.path_hooks and dealing with sys.path_import_cache. It
> > is *key* to PEP 302, and to me simply a killer feature.
> 
> This would be nice, but if the above point is not resolved

Which point?

> and this
> is not concretely implemented, we risk to have meta_path used in
> non-intended ways or be as much problematic as __import__.

The new import hooks operate at *such* a different level that __import__
that I honestly don't see what the problem is.

> Lacking precise policies and clear-cut commitment for its future
> uses, it's better left out (it can be added in the future).

I disagree. Better be able to *use* it and establish conventions based
on actual usage than leave it out and fantasize about what it *could*
do.

PEP 302 should replace 99% of the use cases from __import__ and
hard-coded hooks in import.c. Leaving out sys.meta_path would reduce
this number significantly and would be a really dumb decision.

Just


From just@letterror.com  Sat Dec 28 13:39:01 2002
From: just@letterror.com (Just van Rossum)
Date: Sat, 28 Dec 2002 14:39:01 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <000d01c2ae75$65030aa0$6d94fea9@newmexico>
Message-ID: <r01050400-1022-BBC40DED1A6911D7A7C4003065D5E7E4@[10.0.0.2]>

Samuele Pedroni wrote:

> > Lacking this, it seems equivalent to installing __import__ hooks
> > apart a tiny bit of convenience.
> 
> ok, not irrelevant detail, with meta_path hooks can be removed, still
> where to put a hook in meta_path is problematic.

I have no idea what you're talking about :-(

Just


From guido@python.org  Sat Dec 28 13:48:24 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 28 Dec 2002 08:48:24 -0500
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: Your message of "Sat, 28 Dec 2002 14:37:34 +0100."
 <r01050400-1022-8A8BBD231A6911D7A7C4003065D5E7E4@[10.0.0.2]>
References: <r01050400-1022-8A8BBD231A6911D7A7C4003065D5E7E4@[10.0.0.2]>
Message-ID: <200212281348.gBSDmOH26908@pcp02138704pcs.reston01.va.comcast.net>

> Samuele Pedroni wrote:
> 
> > I agree with Guido that special cookies can be used for normal
> > importers.

[Just]
> I find this rather ugly, and would only do that if it's crucial that the
> pathless importer should be invoked somewhere in the middle of sys.path.
> And I don't have a use case for that. There are plenty of use cases for
> sys.meta_path.

Sure, but pleading for cookies is that they allow easier user control
through PYTHONPATH, and/or through sys.path manipulation; cookies
allow you to control precisely the import order.

It seems they are indeed different features.  Once we have path_hooks,
cookies are a simple consequence; meta_path is an orthogonal feature.

I agree with Samuele's concern that if you have multiple meta_path
importers, ordering them may become an issue.  On the other hand,
ordering of entries in sys.path is a bit of a black art, but in 99% of
the cases, insert-in-front or append-at-end seem suffiecient.  I
expect that the same will be true for sys.meta_path.  (Though it would
be a bit more convenient if indeed it included the built-in, frozen
and path importers -- maybe we can get that done in alpha 2.)

I agree with Just that either of these hooks is *much* more convenient
than overriding __import__ -- because with __import__, you have to
reimplement the whole
look-in-sys.path-try-relative-try-absolute-import-parent-packages
routine, unless you have a truly trivial use (like printing a log
message and calling the real __import__).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From pedronis@bluewin.ch  Sat Dec 28 14:06:56 2002
From: pedronis@bluewin.ch (Samuele Pedroni)
Date: Sat, 28 Dec 2002 15:06:56 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
References: <r01050400-1022-8A8BBD231A6911D7A7C4003065D5E7E4@[10.0.0.2]>  <200212281348.gBSDmOH26908@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <006f01c2ae7a$61ffea80$6d94fea9@newmexico>

From: "Guido van Rossum" <guido@python.org>
> I agree with Samuele's concern that if you have multiple meta_path
> importers, ordering them may become an issue.  On the other hand,
> ordering of entries in sys.path is a bit of a black art, but in 99% of
> the cases, insert-in-front or append-at-end seem suffiecient.  I
> expect that the same will be true for sys.meta_path.  

the difference is that sys.path is really a deployment env issue and there are
PYTHONPATH, site.py etc ...

For meta_path it seems there are programmatical uses (testing frameworks etc),
maybe a priority mechanism is overkill, maybe it would be useful.






From mwh@python.net  Sat Dec 28 14:53:17 2002
From: mwh@python.net (Michael Hudson)
Date: 28 Dec 2002 14:53:17 +0000
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: Just van Rossum's message of "Sat, 28 Dec 2002 14:37:34 +0100"
References: <r01050400-1022-8A8BBD231A6911D7A7C4003065D5E7E4@[10.0.0.2]>
Message-ID: <2m8yya6vqa.fsf@starship.python.net>

Just van Rossum <just@letterror.com> writes:

> I find this rather ugly, and would only do that if it's crucial that the
> pathless importer should be invoked somewhere in the middle of sys.path.
> And I don't have a use case for that. There are plenty of use cases for
> sys.meta_path.

I've not really been following this discussion with utmost care, but
the name "meta_path" doesn't move me.  Wouldn't sys.importers be
better name?  Or maybe I'm misunderstanding it's intent -- which is
still an argument for a better name.

Cheers,
M.

-- 
  Also, remember to put the galaxy back when you've finished, or an
  angry mob of astronomers will come round and kneecap you with a
  small telescope for littering. 
       -- Simon Tatham, ucam.chat, from Owen Dunn's review of the year


From esr@thyrsus.com  Sat Dec 28 16:08:12 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Sat, 28 Dec 2002 11:08:12 -0500
Subject: [Python-Dev] What's the magic formula for regression-testing a single module?
Message-ID: <20021228160812.GA10268@thyrsus.com>

What's the magic formula for regression-testing a single library module?

I found a bug in ConfigParser -- the readfp() and write() functions are
not perfect inverses in some situations.  I think I have a fix, and want
to verify correctness.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From neal@metaslash.com  Sat Dec 28 16:25:22 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Sat, 28 Dec 2002 11:25:22 -0500
Subject: [Python-Dev] What's the magic formula for regression-testing a single module?
In-Reply-To: <20021228160812.GA10268@thyrsus.com>
References: <20021228160812.GA10268@thyrsus.com>
Message-ID: <20021228162522.GY12063@epoch.metaslash.com>

On Sat, Dec 28, 2002 at 11:08:12AM -0500, Eric S. Raymond wrote:
> What's the magic formula for regression-testing a single library module?

I typically do:

        ./python -E -tt ./Lib/test/regrtest.py test_strptime

Replace test_strptime with the test(s) you want, presumably
test_cfgparser.

Neal


From just@letterror.com  Sat Dec 28 17:17:21 2002
From: just@letterror.com (Just van Rossum)
Date: Sat, 28 Dec 2002 18:17:21 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <2m8yya6vqa.fsf@starship.python.net>
Message-ID: <r01050400-1022-40CDBF801A8811D7A7C4003065D5E7E4@[10.0.0.2]>

Michael Hudson wrote:

> I've not really been following this discussion with utmost care, but
> the name "meta_path" doesn't move me.  Wouldn't sys.importers be
> better name?  Or maybe I'm misunderstanding it's intent -- which is
> still an argument for a better name.

I don't like the name much either, but it _does_ fit quite well as the
idea is that sys.path importing gets invoked through an item on
meta_path. Perhaps this sketch helps a little:


sys.meta_path = [
    BuiltinImporter(),  # handles builtin modules
    FrozenImporter(),   # handles frozen modules
    PathImporter(),     # handles sys.path
]


def find_module(fullname, path=None):
    for i in sys.meta_path:
        l = i.find_module(fullname, path)
        if l is not None:
            return l
    return None


class PathImporter:

    def find_module(self, fullname, path=None):
        if path is None:
            path = sys.path
        for p in path:
            i = get_path_importer(p) # deals with sys.path_hooks & cache
            if i is not None:
                l = i.find_module(fullname)
                if l is not None:
                    return l
        retun None


This is how you should look at it, even though it doesn't match the
implementation (yet). Right now you should imagine those three objects
on meta_path as an implicit extension of sys.meta_path, they don't yet
physically exist. This should change in the near future, perhaps even
with 2.3a2.

Just


From python@rcn.com  Sat Dec 28 17:56:56 2002
From: python@rcn.com (python@rcn.com)
Date: Sat, 28 Dec 2002 12:56:56 -0500
Subject: [Python-Dev] Mesenne Twister copyright notice
Message-ID: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net>

During his review of the Mersenne Twister code (in python/nondist/sandbox/twister), Martin raised a question for python-dev about the best way to handle the copyright notice in the original code at http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html .

I incorporated the copyright notice into the head of _random.c, but the notice seems to indicate that it should also be in the documentation (either in librandom.tex or random.py).  I don't think having multiple copies of the copyright notice or having clutter in the LaTeX docs is desirable.  Inclusion by reference or footnote may be sufficient.

Do you guys have any thoughts on the best way to resolve this one?  It is my only unanswered review comment for the module before loading it for the alpha release.


Raymond Hettinger



From python@rcn.com  Sat Dec 28 17:53:41 2002
From: python@rcn.com (Raymond Hettinger)
Date: Sat, 28 Dec 2002 12:53:41 -0500
Subject: [Python-Dev] Mersenne Twister copyright
References: <r01050400-1022-40CDBF801A8811D7A7C4003065D5E7E4@[10.0.0.2]>
Message-ID: <005b01c2ae9a$a19b47a0$125ffea9@oemcomputer>

During his review of the Mersenne Twister code (in python/nondist/sandbox/twister), Martin raised a question for python-dev about
the best way to handle the copyright notice in the original code at http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html .

I incorporated the copyright notice into the head of _random.c, but the notice seems to indicate that it should also be in the
documentation (either in librandom.tex or random.py).  I don't think having multiple copies of the copyright notice or having
clutter in the LaTeX docs is desirable.  Inclusion by reference or footnote may be sufficient.

Do you guys have any thoughts on the best way to resolve this one?  It is my only unanswered review comment for the module before
loading it for the alpha release.


Raymond Hettinger





From mwh@python.net  Sat Dec 28 17:52:43 2002
From: mwh@python.net (Michael Hudson)
Date: 28 Dec 2002 17:52:43 +0000
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: Just van Rossum's message of "Sat, 28 Dec 2002 18:17:21 +0100"
References: <r01050400-1022-40CDBF801A8811D7A7C4003065D5E7E4@[10.0.0.2]>
Message-ID: <2misxeav4k.fsf@starship.python.net>

Just van Rossum <just@letterror.com> writes:

> Michael Hudson wrote:
> 
> > I've not really been following this discussion with utmost care, but
> > the name "meta_path" doesn't move me.  Wouldn't sys.importers be
> > better name?  Or maybe I'm misunderstanding it's intent -- which is
> > still an argument for a better name.
> 
> I don't like the name much either, but it _does_ fit quite well as the
> idea is that sys.path importing gets invoked through an item on
> meta_path. Perhaps this sketch helps a little:
> 
> 
> sys.meta_path = [
>     BuiltinImporter(),  # handles builtin modules
>     FrozenImporter(),   # handles frozen modules
>     PathImporter(),     # handles sys.path
> ]
[...]

Right, that's what I thought was roughly the plan -- but I still don't
like the name.  In this world, sys.path is sort-of subsidiary to
sys.meta_path -- which jars, to me at least.  It makes sense with a
bit of historical knowledge, but not without, IMHO.

Of course, these mild gripes shouldn't delay the real work -- but
names are important.

Cheers,
M.

-- 
  Remember - if all you have is an axe, every problem looks 
  like hours of fun.                                        -- Frossie
               -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html


From mal@lemburg.com  Sat Dec 28 18:19:25 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sat, 28 Dec 2002 19:19:25 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
References: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net>
Message-ID: <3E0DEB2D.9040609@lemburg.com>

python@rcn.com wrote:
> During his review of the Mersenne Twister code (in python/nondist/sandbox/twister), Martin raised a question for python-dev about the best way to handle the copyright notice in the original code at http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html .
> 
> I incorporated the copyright notice into the head of _random.c, but the notice seems to indicate that it should also be in the documentation (either in librandom.tex or random.py).  I don't think having multiple copies of the copyright notice or having clutter in the LaTeX docs is desirable.  Inclusion by reference or footnote may be sufficient.
> 
> Do you guys have any thoughts on the best way to resolve this one?  It is my only unanswered review comment for the module before loading it for the alpha release.

The usual way to deal with this is to include a .txt/.doc file in the
module directory where the module is placed. Examples are the LICENSE.txt
file and the various .doc files we already have in the std lib directory.

I'd suggest to include a twister.txt file together with the module
which then includes the copyright notice.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Sat Dec 28 22:10:21 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 28 Dec 2002 23:10:21 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <3E0DEB2D.9040609@lemburg.com>
References: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net> <3E0DEB2D.9040609@lemburg.com>
Message-ID: <3E0E214D.4060407@v.loewis.de>

M.-A. Lemburg wrote:
> The usual way to deal with this is to include a .txt/.doc file in the
> module directory where the module is placed. Examples are the LICENSE.txt
> file and the various .doc files we already have in the std lib directory.

Can you give a few examples? I could not find a file LICENSE.txt (you 
probably meant LICENSE), and no .txt or .doc file I checked was relevant 
to licensing.

> I'd suggest to include a twister.txt file together with the module
> which then includes the copyright notice.

My concern is that people who produce binary distributions will have to 
include this text file in their distribution as well. People who embed 
Python (in a way that includes _random.c) will have to include the text 
file in their applications.

Do we really want this?

Regards,
Martin





From tim.one@comcast.net  Sat Dec 28 22:26:52 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Dec 2002 17:26:52 -0500
Subject: [Python-Dev] Mersenne Twister copyright
In-Reply-To: <005b01c2ae9a$a19b47a0$125ffea9@oemcomputer>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEPPDEAB.tim.one@comcast.net>

I'd like to see a LICENSES.txt file added to the distribution, into which we
copy the licenses for all the external software we *may* ship with a Python
distribution.  On Windows that's quite a lot (zlib, bz2, Tcl/Tk, OpenSSL,
...).

I'm not even thinking about the legalities here, it's just good manners to
acknowledge other peoples' work.  It's also good manners for people building
on our work to pass on the licenses too, and most licenses formally require
that they do so.

> ...
> Do you guys have any thoughts on the best way to resolve this
> one?

The issue isn't unique to you, so don't feel you have to solve it.

> It is my only unanswered review comment for the module before
> loading it for the alpha release.

I expect to make a few hours to dig into the sandbox code tonight, but don't
let that influence you <wink>.



From mal@lemburg.com  Sat Dec 28 22:58:24 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sat, 28 Dec 2002 23:58:24 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
References: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net>	<3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de>
Message-ID: <3E0E2C90.6060505@lemburg.com>

Martin v. L=F6wis wrote:
> M.-A. Lemburg wrote:
>=20
>> The usual way to deal with this is to include a .txt/.doc file in the
>> module directory where the module is placed. Examples are the LICENSE.=
txt
>> file and the various .doc files we already have in the std lib directo=
ry.
>=20
>=20
> Can you give a few examples? I could not find a file LICENSE.txt (you=20
> probably meant LICENSE), and no .txt or .doc file I checked was relevan=
t=20
> to licensing.

Have a look at the /usr/local/lib/python2.2/ dir. The LICENSE.txt
file has the Python copyright notice.

profile.doc has the license for the profiler.

>> I'd suggest to include a twister.txt file together with the module
>> which then includes the copyright notice.
>=20
> My concern is that people who produce binary distributions will have to=
=20
> include this text file in their distribution as well. People who embed=20
> Python (in a way that includes _random.c) will have to include the text=
=20
> file in their applications.
>=20
> Do we really want this?

If that's what the licenses want, there's no way to drive around
this I guess. Note that people redistributing the code can also
take these .txt files and add them to their application documentation
to meet the license requirements.

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Sat Dec 28 23:13:08 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 29 Dec 2002 00:13:08 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <3E0E2C90.6060505@lemburg.com>
References: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net>	<3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de> <3E0E2C90.6060505@lemburg.com>
Message-ID: <3E0E3004.9000900@v.loewis.de>

M.-A. Lemburg wrote:
> Have a look at the /usr/local/lib/python2.2/ dir. The LICENSE.txt
> file has the Python copyright notice.

Ah, I see. It is renamed to LICENSE.txt when installed.

> profile.doc has the license for the profiler.

Right. It appears that ActivePython is breaking InfoSeeks copyright, by 
not including this file in their distribution...

> If that's what the licenses want, there's no way to drive around
> this I guess. 

There are certainly various options:

a) we could refuse to incorporate the code in question, or postpone that 
until an alternative implementation comes along that does not have such 
requirements, or

b) we could try to contact the authors of the code in question, to find 
out whether they would accept other means of acknowledgement (e.g. 
mentioning their names in the ACKS file, and putting some text into the 
copied source - as Raymond already did)

It is certainly the case that we haven't been very careful with such 
things in the past (in particular in binary distributions), but that is 
IMO no excuse to be more careful now.

Regards,
Martin



From Jack.Jansen@oratrix.com  Sat Dec 28 23:58:49 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Sun, 29 Dec 2002 00:58:49 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <3E0E3004.9000900@v.loewis.de>
Message-ID: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com>

On zondag, dec 29, 2002, at 00:13 Europe/Amsterdam, Martin v. L=F6wis=20
wrote:
> a) we could refuse to incorporate the code in question, or postpone=20
> that until an alternative implementation comes along that does not=20
> have such requirements, or

This sounds like overkill.
>
> b) we could try to contact the authors of the code in question, to=20
> find out whether they would accept other means of acknowledgement=20
> (e.g. mentioning their names in the ACKS file, and putting some text=20=

> into the copied source - as Raymond already did)

This is probably a good idea. Alternatively, there's another option:
c) Add a special directory (Misc/Licenses comes to mind, copy it to=20
lib/python2.3/config/Licenses on install) that has the text files for=20
the various packages that require their license to be in a file of its=20=

own. We could even get all fancy, and install only those licenses for=20
which the corresponding code is actually used.

> It is certainly the case that we haven't been very careful with such=20=

> things in the past (in particular in binary distributions), but that=20=

> is IMO no excuse to be more careful now.

MacPython has always been fairly careful about this, and I wouldn't be=20=

surprised if the same holds for PythonWin.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>       =20
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma=20
Goldman -



From martin@v.loewis.de  Sun Dec 29 00:24:12 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 29 Dec 2002 01:24:12 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com>
References: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com>
Message-ID: <3E0E40AC.6040906@v.loewis.de>

Jack Jansen wrote:
> MacPython has always been fairly careful about this, and I wouldn't be 
> surprised if the same holds for PythonWin.

As Tim points out, the Windows distribution incorporates a number of 
add-on libraries with their own copying conditions, which aren't always 
followed.

For example, the db 1.85 package which was the basis of bsddb on Windows 
still has the original Berkeley advertising clause (which was waived 
now); it also has the requirement that The Regents of the University of 
California are acknowledged and their license terms be included in 
binary-only distributions; I don't think this requirement was followed.

The same holds for other copies of the BSD license, e.g. the one in 
getaddrinfo.c requires that the WIDE project is acknowledged (I'm guilty 
of incorporating that one into Python).

pcre-int.h says that the origin of PCRE must not be misrepresented by 
omission [of Philip Hazel's copyright].

regexpr.c says that Tatu Ylonen's text must appear in all copies.

xmlrpclib.py says that Secret Lab's copyright text must be mentioned in 
documentation.

And so on.

Does MacPython acknowledge all these people and organizations, in the 
documentation?

Regards,
Martin



From mal@lemburg.com  Sun Dec 29 12:33:41 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sun, 29 Dec 2002 13:33:41 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
References: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net>	<3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de>	<3E0E2C90.6060505@lemburg.com> <3E0E3004.9000900@v.loewis.de>
Message-ID: <3E0EEBA5.1020803@lemburg.com>

Martin v. L=F6wis wrote:
> M.-A. Lemburg wrote:
>=20
>> Have a look at the /usr/local/lib/python2.2/ dir. The LICENSE.txt
>> file has the Python copyright notice.
>=20
> Ah, I see. It is renamed to LICENSE.txt when installed.
>=20
>> profile.doc has the license for the profiler.
> =20
> Right. It appears that ActivePython is breaking InfoSeeks copyright, by=
=20
> not including this file in their distribution...

The license is also copied in the documentation for that
module, so I believe everything is OK.

>> If that's what the licenses want, there's no way to drive around
>> this I guess.=20
>=20
> There are certainly various options:
>=20
> a) we could refuse to incorporate the code in question, or postpone tha=
t=20
> until an alternative implementation comes along that does not have such=
=20
> requirements, or

-1

Adding a few license text files to the module directory or
a separate section of the Python documentation shouldn't keep
us from including the code.

> b) we could try to contact the authors of the code in question, to find=
=20
> out whether they would accept other means of acknowledgement (e.g.=20
> mentioning their names in the ACKS file, and putting some text into the=
=20
> copied source - as Raymond already did)

-0

While this would certainly be a much better solution, I don't
think it is required.

Instead, we should come up with a way which satisfies the licenses.
Since most licenses require a copy of the text to be included in
the documentation, I think the easiest and most reliable way of
achieving this is to add a new licenses section to the documentation
and Python's LICENSE file itself (Latex allows including text from
other files in the docs, so adding new licenses to the LICENSE file
would suffice if LICENSE is included verbatim in the Latex docs).

> It is certainly the case that we haven't been very careful with such=20
> things in the past (in particular in binary distributions), but that is=
=20
> IMO no excuse to be more careful now.

Agreed.

Since you have already scanned much of the code, perhaps you
could summarize your license findings in a new sandbox file
until we have come up with a solution that satisfies everyone ?!

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Sun Dec 29 12:43:43 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 29 Dec 2002 13:43:43 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <3E0EEBA5.1020803@lemburg.com>
References: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net>	<3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de>	<3E0E2C90.6060505@lemburg.com> <3E0E3004.9000900@v.loewis.de> <3E0EEBA5.1020803@lemburg.com>
Message-ID: <3E0EEDFF.7020402@v.loewis.de>

M.-A. Lemburg wrote:
> Since you have already scanned much of the code, perhaps you
> could summarize your license findings in a new sandbox file
> until we have come up with a solution that satisfies everyone ?!
> 
That scan was not exhaustive; if anybody would collect this information 
in a systematic manner, that would be much appreciated.

Regards,
Martin




From skip@manatee.mojam.com  Sun Dec 29 13:00:18 2002
From: skip@manatee.mojam.com (Skip Montanaro)
Date: Sun, 29 Dec 2002 07:00:18 -0600
Subject: [Python-Dev] Weekly Python Bug/Patch Summary
Message-ID: <200212291300.gBTD0Irc016025@manatee.mojam.com>

Bug/Patch Summary
-----------------

326 open / 3145 total bugs (+4)
104 open / 1855 total patches (+11)

New Bugs
--------

email: huge address lines blow stack (2002-12-15)
	http://python.org/sf/654362
make errors better like cgi (2002-12-16)
	http://python.org/sf/654558
asyncore.py and "handle_expt" (2002-12-16)
	http://python.org/sf/654766
doctest and exception messages (2002-12-16)
	http://python.org/sf/654783
datetime docs need review, LaTeX (2002-12-16)
	http://python.org/sf/654846
Slightly modify locals() doc? (2002-12-17)
	http://python.org/sf/655271
cPickle not always same as pickle (2002-12-18)
	http://python.org/sf/655802
binascii.a2b_base64 with non base64 data (2002-12-19)
	http://python.org/sf/656392
gettext.py crash on bogus preamble (2002-12-24)
	http://python.org/sf/658233
Accept None for time.ctime() and friends (2002-12-24)
	http://python.org/sf/658254
cPickle does relative import (2002-12-26)
	http://python.org/sf/658693
asyncore connect() and winsock errors (2002-12-26)
	http://python.org/sf/658749
no docs for HTMLParser.handle_pi (2002-12-27)
	http://python.org/sf/659188
'realpath' function missing from os.path (2002-12-27)
	http://python.org/sf/659228
optparse store_true uses 1 and 0 (2002-12-28)
	http://python.org/sf/659604

New Patches
-----------

gzip shouldn't ValueError on corruptfile (2002-12-15)
	http://python.org/sf/654421
Fix bug in IE/CGI [bug 427345] (2002-12-16)
	http://python.org/sf/654910
commands.py for Windows (2002-12-17)
	http://python.org/sf/655421
Add warnings to unsafe Cookie classes (2002-12-18)
	http://python.org/sf/655760
/dev/ptmx support for ptys (cygwin) (2002-12-19)
	http://python.org/sf/656590
Documentation support for PEP 301 (2002-12-23)
	http://python.org/sf/658093
PEP 301 implementation (2002-12-23)
	http://python.org/sf/658094
Mersenne Twister (2002-12-24)
	http://python.org/sf/658251
skips.txt for regrtest.py (2002-12-24)
	http://python.org/sf/658316
Add inet_pton and inet_ntop to socket (2002-12-24)
	http://python.org/sf/658327
email: minimal header encoding (2002-12-24)
	http://python.org/sf/658407
Fix for bug 494589 (2002-12-25)
	http://python.org/sf/658599
regex fixes for _strptime (2002-12-26)
	http://python.org/sf/658820
posixpath missing getctime (2002-12-27)
	http://python.org/sf/658927
Use PyArg_UnpackTuple where possible (2002-12-28)
	http://python.org/sf/659536

Closed Bugs
-----------

Python 2.1 installs shared libs with mode 0700 (2001-05-17)
	http://python.org/sf/425007
Left to right (2001-08-06)
	http://python.org/sf/448679
unreliable file.read() error handling (2002-02-23)
	http://python.org/sf/521782
lib-dynload/*.so wrong permissions (2002-07-17)
	http://python.org/sf/583206
property missing doc'd __name__ attr (2002-09-22)
	http://python.org/sf/612969
missing DECREF's in embedding example (2002-10-21)
	http://python.org/sf/626275
ftplib bails out on empty responses (2002-11-11)
	http://python.org/sf/636685
Undocumented side effect of eval (2002-11-20)
	http://python.org/sf/641111
property() builtin not documented (2002-11-23)
	http://python.org/sf/642742
.extend() doc, .__doc__ too restrictive (2002-11-29)
	http://python.org/sf/645777

Closed Patches
--------------

install lib-dynload .so files mode 555 (2002-10-26)
	http://python.org/sf/629278
Fix for major rexec bugs (2002-11-11)
	http://python.org/sf/636769
Allow keyword args in dict.update() (2002-11-25)
	http://python.org/sf/643565
Set Next Statement for Python debuggers (2002-11-25)
	http://python.org/sf/643835


From max@malva.ua  Sun Dec 29 13:20:38 2002
From: max@malva.ua (Max Ischenko)
Date: Sun, 29 Dec 2002 15:20:38 +0200
Subject: [Python-Dev] Encoding file for KOI8-U
Message-ID: <20021229132038.GA8795@malva.ua>

--zYM0uCDKw75PZbzx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

I wonder, whether koi8-u.py would be included in Python 2.3?
If not, I could submit one. 

Though I'm not the author, it seems correct.

-- 
Bst rgrds, M.A.X.: Mechanical Artificial Xenomorph.

--zYM0uCDKw75PZbzx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="koi8_u.py"

""" Python Character Mapping Codec is modified version of 'koi8_r.py'.
    This character scheme compliant to RFC2319

Written by Marc-Andre Lemburg (mal@lemburg.com).
Modified by Maxim Dzumanenko <mvd@mylinux.com.ua>.

(c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
(c) Copyright 2000 Guido van Rossum.

"""#"

import codecs

### Codec APIs

class Codec(codecs.Codec):

    def encode(self,input,errors='strict'):

        return codecs.charmap_encode(input,errors,encoding_map)
        
    def decode(self,input,errors='strict'):

        return codecs.charmap_decode(input,errors,decoding_map)

class StreamWriter(Codec,codecs.StreamWriter):
    pass
        
class StreamReader(Codec,codecs.StreamReader):
    pass

### encodings module API

def getregentry():

    return (Codec().encode,Codec().decode,StreamReader,StreamWriter)

### Decoding Map

decoding_map = codecs.make_identity_dict(range(256))
decoding_map.update({
	0x0080: 0x2500, #       BOX DRAWINGS LIGHT HORIZONTAL
	0x0081: 0x2502, #       BOX DRAWINGS LIGHT VERTICAL
	0x0082: 0x250C, #       BOX DRAWINGS LIGHT DOWN AND RIGHT
	0x0083: 0x2510, #       BOX DRAWINGS LIGHT DOWN AND LEFT
	0x0084: 0x2514, #       BOX DRAWINGS LIGHT UP AND RIGHT
	0x0085: 0x2518, #       BOX DRAWINGS LIGHT UP AND LEFT
	0x0086: 0x251C, #       BOX DRAWINGS LIGHT VERTICAL AND RIGHT
	0x0087: 0x2524, #       BOX DRAWINGS LIGHT VERTICAL AND LEFT
	0x0088: 0x252C, #       BOX DRAWINGS LIGHT DOWN AND HORIZONTAL
	0x0089: 0x2534, #       BOX DRAWINGS LIGHT UP AND HORIZONTAL
	0x008a: 0x253C, #       BOX DRAWINGS LIGHT VERTICAL AND HORIZONTAL
	0x008b: 0x2580, #       UPPER HALF BLOCK
	0x008c: 0x2584, #       LOWER HALF BLOCK
	0x008d: 0x2588, #       FULL BLOCK
	0x008e: 0x258C, #       LEFT HALF BLOCK
	0x008f: 0x2590, #       RIGHT HALF BLOCK
	0x0090: 0x2591, #       LIGHT SHADE
	0x0091: 0x2592, #       MEDIUM SHADE
	0x0092: 0x2593, #       DARK SHADE
	0x0093: 0x2320, #       TOP HALF INTEGRAL
	0x0094: 0x25A0, #       BLACK SQUARE
	0x0095: 0x2219, #       BULLET OPERATOR
	0x0096: 0x221A, #       SQUARE ROOT
	0x0097: 0x2248, #       ALMOST EQUAL TO
	0x0098: 0x2264, #       LESS THAN OR EQUAL TO
	0x0099: 0x2265, #       GREATER THAN OR EQUAL TO
	0x009a: 0x00A0, #       NO-BREAK SPACE
	0x009b: 0x2321, #       BOTTOM HALF INTEGRAL
	0x009c: 0x00B0, #       DEGREE SIGN
	0x009d: 0x00B2, #       SUPERSCRIPT DIGIT TWO
	0x009e: 0x00B7, #       MIDDLE DOT
	0x009f: 0x00F7, #       DIVISION SIGN
	0x00a0: 0x2550, #       BOX DRAWINGS DOUBLE HORIZONTAL
	0x00a1: 0x2551, #       BOX DRAWINGS DOUBLE VERTICAL
	0x00a2: 0x2552, #       BOX DRAWINGS DOWN SINGLE AND RIGHT DOUBLE
	0x00a3: 0x0451, #       CYRILLIC SMALL LETTER IO
	0x00a4: 0x0454, #       CYRILLIC SMALL LETTER UKRAINIAN IE
	0x00a5: 0x2554, #       BOX DRAWINGS DOUBLE DOWN AND RIGHT
	0x00a6: 0x0456, #       CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I
	0x00a7: 0x0457, #       CYRILLIC SMALL LETTER YI (Ukrainian)
	0x00a8: 0x2557, #       BOX DRAWINGS DOUBLE DOWN AND LEFT
	0x00a9: 0x2558, #       BOX DRAWINGS UP SINGLE AND RIGHT DOUBLE
	0x00aa: 0x2559, #       BOX DRAWINGS UP DOUBLE AND RIGHT SINGLE
	0x00ab: 0x255A, #       BOX DRAWINGS DOUBLE UP AND RIGHT
	0x00ac: 0x255B, #       BOX DRAWINGS UP SINGLE AND LEFT DOUBLE
	0x00ad: 0x0491, #       CYRILLIC SMALL LETTER GHE WITH UPTURN
	0x00ae: 0x255D, #       BOX DRAWINGS DOUBLE UP AND LEFT
	0x00af: 0x255E, #       BOX DRAWINGS VERTICAL SINGLE AND RIGHT DOUBLE
	0x00b0: 0x255F, #       BOX DRAWINGS VERTICAL DOUBLE AND RIGHT SINGLE
	0x00b1: 0x2560, #       BOX DRAWINGS DOUBLE VERTICAL AND RIGHT
	0x00b2: 0x2561, #       BOX DRAWINGS VERTICAL SINGLE AND LEFT DOUBLE
	0x00b3: 0x0401, #       CYRILLIC CAPITAL LETTER IO
	0x00b4: 0x0404, #       CYRILLIC CAPITAL LETTER UKRAINIAN IE
	0x00b5: 0x2563, #       DOUBLE VERTICAL AND LEFT
	0x00b6: 0x0406, #       CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I
	0x00b7: 0x0407, #       CYRILLIC CAPITAL LETTER YI (Ukrainian)
	0x00b8: 0x2566, #       BOX DRAWINGS DOUBLE DOWN AND HORIZONTAL
	0x00b9: 0x2567, #       BOX DRAWINGS UP SINGLE AND HORIZONTAL DOUBLE
	0x00ba: 0x2568, #       BOX DRAWINGS UP DOUBLE AND HORIZONTAL SINGLE
	0x00bb: 0x2569, #       BOX DRAWINGS DOUBLE UP AND HORIZONTAL
	0x00bc: 0x256A, #       BOX DRAWINGS VERTICAL SINGLE AND HORIZONTAL DOUBLE
	0x00bd: 0x0490, #       CYRILLIC CAPITAL LETTER GHE WITH UPTURN
	0x00be: 0x256C, #       BOX DRAWINGS DOUBLE VERTICAL AND HORIZONTAL
	0x00bf: 0x00A9, #       COPYRIGHT SIGN
	0x00c0: 0x044E, #       CYRILLIC SMALL LETTER YU
	0x00c1: 0x0430, #       CYRILLIC SMALL LETTER A
	0x00c2: 0x0431, #       CYRILLIC SMALL LETTER BE
	0x00c3: 0x0446, #       CYRILLIC SMALL LETTER TSE
	0x00c4: 0x0434, #       CYRILLIC SMALL LETTER DE
	0x00c5: 0x0435, #       CYRILLIC SMALL LETTER IE
	0x00c6: 0x0444, #       CYRILLIC SMALL LETTER EF
	0x00c7: 0x0433, #       CYRILLIC SMALL LETTER GHE
	0x00c8: 0x0445, #       CYRILLIC SMALL LETTER HA
	0x00c9: 0x0438, #       CYRILLIC SMALL LETTER I
	0x00ca: 0x0439, #       CYRILLIC SMALL LETTER SHORT I
	0x00cb: 0x043A, #       CYRILLIC SMALL LETTER KA
	0x00cc: 0x043B, #       CYRILLIC SMALL LETTER EL
	0x00cd: 0x043C, #       CYRILLIC SMALL LETTER EM
	0x00ce: 0x043D, #       CYRILLIC SMALL LETTER EN
	0x00cf: 0x043E, #       CYRILLIC SMALL LETTER O
	0x00d0: 0x043F, #       CYRILLIC SMALL LETTER PE
	0x00d1: 0x044F, #       CYRILLIC SMALL LETTER YA
	0x00d2: 0x0440, #       CYRILLIC SMALL LETTER ER
	0x00d3: 0x0441, #       CYRILLIC SMALL LETTER ES
	0x00d4: 0x0442, #       CYRILLIC SMALL LETTER TE
	0x00d5: 0x0443, #       CYRILLIC SMALL LETTER U
	0x00d6: 0x0436, #       CYRILLIC SMALL LETTER ZHE
	0x00d7: 0x0432, #       CYRILLIC SMALL LETTER VE
	0x00d8: 0x044C, #       CYRILLIC SMALL LETTER SOFT SIGN
	0x00d9: 0x044B, #       CYRILLIC SMALL LETTER YERU
	0x00da: 0x0437, #       CYRILLIC SMALL LETTER ZE
	0x00db: 0x0448, #       CYRILLIC SMALL LETTER SHA
	0x00dc: 0x044D, #       CYRILLIC SMALL LETTER E
	0x00dd: 0x0449, #       CYRILLIC SMALL LETTER SHCHA
	0x00de: 0x0447, #       CYRILLIC SMALL LETTER CHE
	0x00df: 0x044A, #       CYRILLIC SMALL LETTER HARD SIGN
	0x00e0: 0x042E, #       CYRILLIC CAPITAL LETTER YU
	0x00e1: 0x0410, #       CYRILLIC CAPITAL LETTER A
	0x00e2: 0x0411, #       CYRILLIC CAPITAL LETTER BE
	0x00e3: 0x0426, #       CYRILLIC CAPITAL LETTER TSE
	0x00e4: 0x0414, #       CYRILLIC CAPITAL LETTER DE
	0x00e5: 0x0415, #       CYRILLIC CAPITAL LETTER IE
	0x00e6: 0x0424, #       CYRILLIC CAPITAL LETTER EF
	0x00e7: 0x0413, #       CYRILLIC CAPITAL LETTER GHE
	0x00e8: 0x0425, #       CYRILLIC CAPITAL LETTER HA
	0x00e9: 0x0418, #       CYRILLIC CAPITAL LETTER I
	0x00ea: 0x0419, #       CYRILLIC CAPITAL LETTER SHORT I
	0x00eb: 0x041A, #       CYRILLIC CAPITAL LETTER KA
	0x00ec: 0x041B, #       CYRILLIC CAPITAL LETTER EL
	0x00ed: 0x041C, #       CYRILLIC CAPITAL LETTER EM
	0x00ee: 0x041D, #       CYRILLIC CAPITAL LETTER EN
	0x00ef: 0x041E, #       CYRILLIC CAPITAL LETTER O
	0x00f0: 0x041F, #       CYRILLIC CAPITAL LETTER PE
	0x00f1: 0x042F, #       CYRILLIC CAPITAL LETTER YA
	0x00f2: 0x0420, #       CYRILLIC CAPITAL LETTER ER
	0x00f3: 0x0421, #       CYRILLIC CAPITAL LETTER ES
	0x00f4: 0x0422, #       CYRILLIC CAPITAL LETTER TE
	0x00f5: 0x0423, #       CYRILLIC CAPITAL LETTER U
	0x00f6: 0x0416, #       CYRILLIC CAPITAL LETTER ZHE
	0x00f7: 0x0412, #       CYRILLIC CAPITAL LETTER VE
	0x00f8: 0x042C, #       CYRILLIC CAPITAL LETTER SOFT SIGN
	0x00f9: 0x042B, #       CYRILLIC CAPITAL LETTER YERU
	0x00fa: 0x0417, #       CYRILLIC CAPITAL LETTER ZE
	0x00fb: 0x0428, #       CYRILLIC CAPITAL LETTER SHA
	0x00fc: 0x042D, #       CYRILLIC CAPITAL LETTER E
	0x00fd: 0x0429, #       CYRILLIC CAPITAL LETTER SHCHA
	0x00fe: 0x0427, #       CYRILLIC CAPITAL LETTER CHE
	0x00ff: 0x042A, #       CYRILLIC CAPITAL LETTER HARD SIGN
})

### Encoding Map

encoding_map = codecs.make_encoding_map(decoding_map)

--zYM0uCDKw75PZbzx--


From mal@lemburg.com  Sun Dec 29 14:40:48 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sun, 29 Dec 2002 15:40:48 +0100
Subject: [Python-Dev] Encoding file for KOI8-U
References: <20021229132038.GA8795@malva.ua>
Message-ID: <3E0F0970.6060104@lemburg.com>

Max Ischenko wrote:
> I wonder, whether koi8-u.py would be included in Python 2.3?
> If not, I could submit one. 

Such a codec is already included in Python CVS.

> Though I'm not the author, it seems correct.

Maxim submitted this codec a while ago. It has a few bugs. The
version in CVS is compliant to the RFC and also faster:

""" Python Character Mapping Codec for KOI8U.

     This character scheme is compliant to RFC2319

Written by Marc-Andre Lemburg (mal@lemburg.com).
Modified by Maxim Dzumanenko <mvd@mylinux.com.ua>.

(c) Copyright 2002, Python Software Foundation.

"""#"

import codecs, koi8_r

### Codec APIs

class Codec(codecs.Codec):

     def encode(self,input,errors='strict'):

         return codecs.charmap_encode(input,errors,encoding_map)

     def decode(self,input,errors='strict'):

         return codecs.charmap_decode(input,errors,decoding_map)

class StreamWriter(Codec,codecs.StreamWriter):
     pass

class StreamReader(Codec,codecs.StreamReader):
     pass

### encodings module API

def getregentry():

     return (Codec().encode,Codec().decode,StreamReader,StreamWriter)

### Decoding Map

decoding_map = koi8_r.decoding_map.copy()
decoding_map.update({
         0x00a4: 0x0454, #       CYRILLIC SMALL LETTER UKRAINIAN IE
         0x00a6: 0x0456, #       CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I
         0x00a7: 0x0457, #       CYRILLIC SMALL LETTER YI (UKRAINIAN)
         0x00ad: 0x0491, #       CYRILLIC SMALL LETTER UKRAINIAN GHE WITH UPTURN
         0x00b4: 0x0403, #       CYRILLIC CAPITAL LETTER UKRAINIAN IE
         0x00b6: 0x0406, #       CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I
         0x00b7: 0x0407, #       CYRILLIC CAPITAL LETTER YI (UKRAINIAN)
         0x00bd: 0x0490, #       CYRILLIC CAPITAL LETTER UKRAINIAN GHE WITH UPTURN
})

### Encoding Map

encoding_map = codecs.make_encoding_map(decoding_map)

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From python@rcn.com  Sun Dec 29 17:18:34 2002
From: python@rcn.com (Raymond Hettinger)
Date: Sun, 29 Dec 2002 12:18:34 -0500
Subject: [Python-Dev] Mesenne Twister copyright notice
References: <E18SLCi-000784-00@smtp02.mrf.mail.rcn.net>	<3E0DEB2D.9040609@lemburg.com> <3E0E214D.4060407@v.loewis.de>	<3E0E2C90.6060505@lemburg.com> <3E0E3004.9000900@v.loewis.de> <3E0EEBA5.1020803@lemburg.com>
Message-ID: <008001c2af5e$52bb7a00$7311a044@oemcomputer>

[MAL]
> Instead, we should come up with a way which satisfies the licenses.
> Since most licenses require a copy of the text to be included in
> the documentation, I think the easiest and most reliable way of
> achieving this is to add a new licenses section to the documentation
> and Python's LICENSE file itself (Latex allows including text from
> other files in the docs, so adding new licenses to the LICENSE file
> would suffice if LICENSE is included verbatim in the Latex docs).

This seems to be the most straight-forward way to go.
If everyone agrees, I'll start the ball rolling by adding the
verbatim text for the MersenneTwister code.


Raymond Hettinger





From Jack.Jansen@oratrix.com  Sun Dec 29 21:39:10 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Sun, 29 Dec 2002 22:39:10 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <3E0E40AC.6040906@v.loewis.de>
Message-ID: <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>

On zondag, dec 29, 2002, at 01:24 Europe/Amsterdam, Martin v. L=F6wis=20
wrote:
>
> Does MacPython acknowledge all these people and organizations, in the=20=

> documentation?

Oops, you're right: MacPython acknowledges all the stuff that *I*=20
added, but of course I never thought about the things in the core:-(

On the other thread: will including the copyright=20
notices/acknowledgements in the documentation be good enough if we=20
don't always ship the documentation? Or should we put this in a special=20=

section that is always shipped, even with binary installers that don't=20=

include documentation?
--
- Jack Jansen        <Jack.Jansen@oratrix.com>       =20
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma=20
Goldman -



From martin@v.loewis.de  Sun Dec 29 21:56:40 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 29 Dec 2002 22:56:40 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
References: <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
Message-ID: <3E0F6F98.1000404@v.loewis.de>

Jack Jansen wrote:
> On the other thread: will including the copyright 
> notices/acknowledgements in the documentation be good enough if we don't 
> always ship the documentation? 

This is exactly my concern. It is essentially for people producing 
binary distributions to decide how they want this to be organized.

> Or should we put this in a special 
> section that is always shipped, even with binary installers that don't 
> include documentation?

For the specific case in question, I think MAL's suggestion of adding 
yet another file is sufficient - maintainers of binary distributions 
then need to collect all those text files and distribute them. Complying 
  with the various licenses is really something that the packagers most 
solve for themselves - the source distribution can only prepare things 
in a way that makes complying simple.

Of course, if the license says that you must have certain words in the 
documentation of a binary distribution, then you really must have those 
words somewhere: either in text files that you "declare" to be 
documentation, or (probably more to the spirit of these clauses) in a 
place where users of the binary distribution won't overlook them.

My feeling is that this can get quite annoying both for people producing 
binary distributions, as well as for people using them. So I would 
rather negotiate terms with the authors, or consider rejection of 
contributions that have such clauses. Of course, if nobody but me is 
annoyed by such clauses, I won't fight windmills.

Regards,
Martin





From Jack.Jansen@oratrix.com  Sun Dec 29 23:03:09 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Mon, 30 Dec 2002 00:03:09 +0100
Subject: [Python-Dev] Moving Mac/Lib to Lib/plat-mac before 2.3a1?
Message-ID: <B2E5063E-1B81-11D7-AC45-003065517236@oratrix.com>

Oops, there something that's been on my todo list for quite a while but 
that I haven't taken any action on (at least I can put part of the 
blame on Just, it was his idea and he didn't take any action either:-): 
moving Mac/Lib to Lib/plat-mac. The main reason for doing this is that 
most (if not all) of those modules are useable in unix-Python on MacOSX 
too, and moreover it would make the MacPython-OS9 tree look a little 
more like the normal Python tree. site.py would get one extra line with 
a special case (if sys.platform==darwin we not only add plat-darwin to 
the path but also plat-mac).

Is it a good idea to try and get this in before 2.3a1? Is it a better 
idea to try and get this in between 2.3a1 and the next release (2.3a2? 
2.3b1?)?

And on the technical implementation: I think I would ask the SF 
maintainers to *copy* Mac/Lib to Lib/plat-mac, and then manually delete 
Mac/Lib on the trunk and Lib/plat-mac on the 2.2 maintenance branch. Is 
that a good idea?

A similar case can be made for other Mac subdirs (Mac/Tools->Tools/Mac, 
Mac/scripts->Tools/Mac/scripts), but that is less essential and can 
probably wait until later.
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From skip@pobox.com  Sun Dec 29 23:23:07 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sun, 29 Dec 2002 17:23:07 -0600
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
References: <3E0E40AC.6040906@v.loewis.de>
 <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
Message-ID: <15887.33755.211400.175809@montanaro.dyndns.org>

    Jack> On the other thread: will including the copyright
    Jack> notices/acknowledgements in the documentation be good enough if we
    Jack> don't always ship the documentation? 

Is there any reason the copyright builtin can't be embellished to spit out
all the necessary copyrights?

Skip


From lalo@laranja.org  Mon Dec 30 00:09:09 2002
From: lalo@laranja.org (Lalo Martins)
Date: Sun, 29 Dec 2002 22:09:09 -0200
Subject: [Python-Dev] Proposed contribution to the library: PAX (another XML library)
Message-ID: <20021230000909.GC15340@laranja.org>

PAX - the "Pythonic API for XML" - is a XML-handling library.

Like with DOM, you're supposed to first parse the whole stream into the
internal representation, and then operate on it.

What distinguishes PAX is that its implementation is python-oriented; you
iterate trough child nodes via the normal ways (for, getitem, and in 2.2+,
an iterator), the attributes are dictionary-based, etc.

I used it to successfully reimplement TAL and PageTemplates (used in Zope).
This reimplementation (AltPT) is quite stable and in production users in a
few dozen servers around the world.

PAX also includes a powerful python-oriented transform engine, which I used
in Alt-TAL.

The code has been tested on Python 2.1, 2.2 and recent CVS checkouts.  It
not only works, but transparently uses recent interesting features when
available - notably, enumerate().

Most of the code is in public domain, and of course I make no objections to
switching to PSL.  In fact the package was designed all along with the
intention of offering it to the standard library (and I like to think that
this led me to cleaner design and a readable implementation).

It is in active maintenence; if not much development happened in the last
few months, it's because it reached some maturity.  In fact I just released
1.0alpha1 before sending this mail (15k tar.bz2)

PAX has a small site at my very-slow (due to bad connectivity) Zope server
at http://python.laranja.org/pax/ - this site, btw, runs on AltPT.  The
releases can be found there, and the cvs is (for hystorical reasons) at
:pserver:anonymous@cvs.sf.net:/cvsroot/flyingcircus
module miscfiles/python/pax (because it was originally developed as I needed
an XML library for my "flyingcircus" pda project).

Should I write a PEP?  Or a patch (silly, it would be a quite big patch)?

[]s,
                                               |alo
                                               +----
--
            Those who trade freedom for security
               lose both and deserve neither.
--
http://www.laranja.org/                mailto:lalo@laranja.org
         pgp key: http://www.laranja.org/pessoal/pgp

Eu jogo RPG! (I play RPG)         http://www.eujogorpg.com.br/
GNU: never give up freedom                 http://www.gnu.org/


From guido@python.org  Mon Dec 30 00:17:36 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 29 Dec 2002 19:17:36 -0500
Subject: [Python-Dev] Moving Mac/Lib to Lib/plat-mac before 2.3a1?
In-Reply-To: Your message of "Mon, 30 Dec 2002 00:03:09 +0100."
 <B2E5063E-1B81-11D7-AC45-003065517236@oratrix.com>
References: <B2E5063E-1B81-11D7-AC45-003065517236@oratrix.com>
Message-ID: <200212300017.gBU0Ha012546@pcp02138704pcs.reston01.va.comcast.net>

> Oops, there something that's been on my todo list for quite a while but 
> that I haven't taken any action on (at least I can put part of the 
> blame on Just, it was his idea and he didn't take any action either:-): 
> moving Mac/Lib to Lib/plat-mac. The main reason for doing this is that 
> most (if not all) of those modules are useable in unix-Python on MacOSX 
> too, and moreover it would make the MacPython-OS9 tree look a little 
> more like the normal Python tree. site.py would get one extra line with 
> a special case (if sys.platform==darwin we not only add plat-darwin to 
> the path but also plat-mac).
> 
> Is it a good idea to try and get this in before 2.3a1? Is it a better 
> idea to try and get this in between 2.3a1 and the next release (2.3a2? 
> 2.3b1?)?

Either way works for me -- I have no problem with doing it now.

> And on the technical implementation: I think I would ask the SF 
> maintainers to *copy* Mac/Lib to Lib/plat-mac, and then manually delete 
> Mac/Lib on the trunk and Lib/plat-mac on the 2.2 maintenance branch. Is 
> that a good idea?

It's sort of history falsification -- and you'd have to delete it on
the 2.1 branch too (we might issue 2.1.4).

I'm also a bit worried that the SF folks don't respond in time for the
release.  Are you sure you need the CVS history in the new location?

> A similar case can be made for other Mac subdirs (Mac/Tools->Tools/Mac, 
> Mac/scripts->Tools/Mac/scripts), but that is less essential and can 
> probably wait until later.

OK.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From dave@boost-consulting.com  Mon Dec 30 01:29:19 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Sun, 29 Dec 2002 20:29:19 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
Message-ID: <ufzsgi9ao.fsf@boost-consulting.com>

Recently an issue has come up on the C++-sig which I think merits a
little attention here.  To boil it down, the situation looks like
this:

Shared library Q uses threading but not Python.  It supplies a an
interface by which users can supply callback functions.  Some of these
callbacks will be invoked directly in response to external calls into
Q; others will be invoked on threads started by calls into Q.

Python extension module A calls shared library Q, but doesn't use its
callback interface.  It works fine by itself.

Python extension module B calls shared library Q and uses Q's callback
interface.  Because some of the callbacks need to use the Python API,
and *might* be invoked by threads, they must all acquire the GIL.
Because they also might be invoked by direct calls into Q, B must
always release the GIL before calling anything in Q.

Problem: using B while A is loaded breaks A: because B has installed
callbacks in Q that acquire the GIL, A must also release the GIL
before calling into Q.

Notice that the author of A may have had no reason to believe anyone
would install Python callbacks in Q!

It seems to me that for situations like these, where a function may or
may not be called on Python's main thread, it would be helpful if
Python supplied a "recursive mutex" GIL acquisition/release pair, for
which acquisition and release on the main thread would simply bump a
counter.  Is this something that was considered and rejected?

TIA,
Dave

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From pyth@devel.trillke.net  Mon Dec 30 01:43:06 2002
From: pyth@devel.trillke.net (holger krekel)
Date: Mon, 30 Dec 2002 02:43:06 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <15887.33755.211400.175809@montanaro.dyndns.org>; from skip@pobox.com on Sun, Dec 29, 2002 at 05:23:07PM -0600
References: <3E0E40AC.6040906@v.loewis.de> <F7288690-1B75-11D7-AC45-003065517236@oratrix.com> <15887.33755.211400.175809@montanaro.dyndns.org>
Message-ID: <20021230024306.A2841@prim.han.de>

Skip Montanaro wrote:
> 
>     Jack> On the other thread: will including the copyright
>     Jack> notices/acknowledgements in the documentation be good enough if we
>     Jack> don't always ship the documentation? 
> 
> Is there any reason the copyright builtin can't be embellished to spit out
> all the necessary copyrights?

great idea. let's implement drm.py. That should finally convince 
the companies in the content industry to embrace python. 

    "of course, we have DRM builtin into python."

almost-as-good-as-region-codes-ly y's holger


From nhodgson@bigpond.net.au  Mon Dec 30 02:12:07 2002
From: nhodgson@bigpond.net.au (Neil Hodgson)
Date: Mon, 30 Dec 2002 13:12:07 +1100
Subject: [Python-Dev] Mesenne Twister copyright notice
References: <4F26502E-1AC0-11D7-951C-003065517236@oratrix.com> <3E0E40AC.6040906@v.loewis.de>
Message-ID: <01d401c2afa8$da7d0c70$3da48490@neil>

Martin v. Löwis:

> pcre-int.h says that the origin of PCRE must not be misrepresented by
> omission [of Philip Hazel's copyright].

   Discussion of this requirement for Scintilla (where I won't accept
contributions with attribution requirements) led to this from Philip Hazel:

% Yes. The licence says you must acknowledge PCRE in any software you
% distribute that includes it. So Scintilla should acknowledge it.
% However, the licence does not say that you have to impose this condition
% on anybody who subsequently distrubutes other software that includes
% your software. At least, that's one way I can read the licence.
%
% I'll put a sentence in if I can find some wording that says that
% reasonably concisely.

   However, the current PCRE license still reads to me to require
attribution no matter how deeply embedded it is.
http://www.pcre.org/license.txt

   Neil



From barry@python.org  Mon Dec 30 02:22:36 2002
From: barry@python.org (Barry A. Warsaw)
Date: Sun, 29 Dec 2002 21:22:36 -0500
Subject: [Python-Dev] Moving Mac/Lib to Lib/plat-mac before 2.3a1?
References: <B2E5063E-1B81-11D7-AC45-003065517236@oratrix.com>
Message-ID: <15887.44524.384976.546553@gargle.gargle.HOWL>

>>>>> "JJ" == Jack Jansen <Jack.Jansen@oratrix.com> writes:

    JJ> And on the technical implementation: I think I would ask the
    JJ> SF maintainers to *copy* Mac/Lib to Lib/plat-mac, and then
    JJ> manually delete Mac/Lib on the trunk and Lib/plat-mac on the
    JJ> 2.2 maintenance branch. Is that a good idea?

My suggestion would be to grab the a copy of the cvs repository, do
the re-organization yourself, and send SF a tarball of just the new
directories and instructions for where you want them to unpack it.
Then cvsrm all the individual files and let cvs up -P zap the old
directories.

-Barry



From neal@metaslash.com  Mon Dec 30 03:04:06 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Sun, 29 Dec 2002 22:04:06 -0500
Subject: [Python-Dev] test_pyclbr failling
Message-ID: <20021230030406.GB12063@epoch.metaslash.com>

test_pyclbr is failing due to the changes in random:

test_pyclbr
l1=['Random']
l2=['CoreGenerator']
ignore=('_verify',)
class=<class 'random.Random'>
test test_pyclbr failed -- Traceback (most recent call last):
  File "/home/neal/build/python/dist/src/Lib/test/test_pyclbr.py", line 146, in test_others
    cm('random', ignore=('_verify',)) # deleted
  File "/home/neal/build/python/dist/src/Lib/test/test_pyclbr.py", line 97, in checkModule
    self.assertListEq(real_bases, pyclbr_bases, ignore)
  File "/home/neal/build/python/dist/src/Lib/test/test_pyclbr.py", line 33, in assertListEq
    self.fail("%r missing" % item)
  File "/home/neal/build/python/dist/src/Lib/unittest.py", line 260, in fail
    raise self.failureException, msg
AssertionError: 'Random' missing

Neal


From aahz@pythoncraft.com  Mon Dec 30 03:45:52 2002
From: aahz@pythoncraft.com (Aahz)
Date: Sun, 29 Dec 2002 22:45:52 -0500
Subject: [Python-Dev] GC at exit?
Message-ID: <20021230034551.GA18622@panix.com>

[Originally posted to comp.lang.python with no response; asking here
before filing a bug report]

Is garbage collection supposed to run when Python exits?  The following
program does not print any output, unless I uncomment the gc.collect()
(or add a for loop that forces GC after creating the cycle):

import gc

class A:
    pass

class B:
    def __del__(self):
        print "Bye-bye!"

a = A()
b = A()
a.b = b
b.a = a
a.x = B()

del a
del b

#gc.collect()
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I disrespectfully agree."  --SJM


From tim.one@comcast.net  Mon Dec 30 04:45:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Dec 2002 23:45:56 -0500
Subject: [Python-Dev] GC at exit?
In-Reply-To: <20021230034551.GA18622@panix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEDFDFAB.tim.one@comcast.net>

[Aahz]
> [Originally posted to comp.lang.python with no response; asking here
> before filing a bug report]
>
> Is garbage collection supposed to run when Python exits?

The language reference manual doesn't promise that any garbage will be
collected, ever.  So, no, "supposed to" doesn't apply.



From mmclay@comcast.net  Mon Dec 30 04:49:42 2002
From: mmclay@comcast.net (Michael McLay)
Date: Sun, 29 Dec 2002 23:49:42 -0500
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <15887.33755.211400.175809@montanaro.dyndns.org>
References: <3E0E40AC.6040906@v.loewis.de>
 <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
 <15887.33755.211400.175809@montanaro.dyndns.org>
Message-ID: <200212292349.29240.mclay@nist.gov>

On Sunday 29 December 2002 06:23 pm, Skip Montanaro wrote:
>     Jack> On the other thread: will including the copyright
>     Jack> notices/acknowledgements in the documentation be good enough if
> we Jack> don't always ship the documentation?
>
> Is there any reason the copyright builtin can't be embellished to spit out
> all the necessary copyrights?

A requirement could be added to the PEP 2 that would require a copyright 
disclosure be included in the package/module so that the name "copyright" 
would retrieve the relevant copyright for that package/module. A copyright 
that did not require disclosure would return None.

(BTW, PEP two is titled "Procedure for Adding New Modules". Should that be 
renamed "Procedure for Adding New Modules and Packages"?)

>>> import mycoolpackage
>>> mycoolpackage.copyright
'All rights to the use of mycoolpackage are hereby granted to the PSF. \n\n    
enjoy  - authors of mycoolpackage\n'
>>>

The copyright builtin could be extended to add a search of all imported 
libraries to retrieve all of the copyrights that are used in an application.




From skip@pobox.com  Mon Dec 30 05:00:08 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sun, 29 Dec 2002 23:00:08 -0600
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <200212292349.29240.mclay@nist.gov>
References: <3E0E40AC.6040906@v.loewis.de>
 <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
 <15887.33755.211400.175809@montanaro.dyndns.org>
 <200212292349.29240.mclay@nist.gov>
Message-ID: <15887.53976.949262.429193@montanaro.dyndns.org>

    Michael> A requirement could be added to the PEP 2 that would require a
    Michael> copyright disclosure be included in the package/module so that
    Michael> the name "copyright" would retrieve the relevant copyright for
    Michael> that package/module.

Excellent idea.  I would name it "__copyright__" though.

Skip


From jeremy@alum.mit.edu  Mon Dec 30 05:04:36 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 30 Dec 2002 00:04:36 -0500
Subject: [Python-Dev] test_pyclbr failling
In-Reply-To: <20021230030406.GB12063@epoch.metaslash.com>
References: <20021230030406.GB12063@epoch.metaslash.com>
Message-ID: <15887.54244.866974.808863@slothrop.zope.com>

It would be helpful if the pyclbr tests could be re-written so that
they don't depend on an arbitrary set of unrelated modules.  While
people should run the full test suite, it seems common to run a
limited set of tests when you're changing a single module and
test_pyclbr always gets skipped.

(I also wonder how useful this pyclbr module is for modern Python,
since almost every module in test_pyclbr that has been modified in the
last year has caused tests to fail.)

Jeremy



From python@rcn.com  Mon Dec 30 06:00:26 2002
From: python@rcn.com (Raymond Hettinger)
Date: Mon, 30 Dec 2002 01:00:26 -0500
Subject: [Python-Dev] Mesenne Twister copyright notice
References: <3E0E40AC.6040906@v.loewis.de> <F7288690-1B75-11D7-AC45-003065517236@oratrix.com> <15887.33755.211400.175809@montanaro.dyndns.org> <200212292349.29240.mclay@nist.gov>
Message-ID: <008201c2afc8$c0659d00$125ffea9@oemcomputer>

> A requirement could be added to the PEP 2 that would require a copyright 
> disclosure be included in the package/module so that the name "copyright" 
> would retrieve the relevant copyright for that package/module. A copyright 
> that did not require disclosure would return None.

Though it makes the copyright accessible in a binary distribution, I don't
think it fulfills the license requirements of being in printed in the
documentation.

So far, the simplest thing that gets the job done and minimizes the PITA 
factor is to have a section in the docs for licenses, paste them in a
verbatim block, and make a link to them from the module docs.


Raymond Hettinger

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################


From skip@pobox.com  Mon Dec 30 06:11:37 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 30 Dec 2002 00:11:37 -0600
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <008201c2afc8$c0659d00$125ffea9@oemcomputer>
References: <3E0E40AC.6040906@v.loewis.de>
 <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
 <15887.33755.211400.175809@montanaro.dyndns.org>
 <200212292349.29240.mclay@nist.gov>
 <008201c2afc8$c0659d00$125ffea9@oemcomputer>
Message-ID: <15887.58265.41467.93079@montanaro.dyndns.org>

    >> A requirement could be added to the PEP 2 that would require a
    >> copyright disclosure be included in the package/module so that the
    >> name "copyright" would retrieve the relevant copyright for that
    >> package/module. A copyright that did not require disclosure would
    >> return None.

    Raymond> Though it makes the copyright accessible in a binary
    Raymond> distribution, I don't think it fulfills the license
    Raymond> requirements of being in printed in the documentation.

Ah, but now you have a "machine" that will generate the bit of documentation
you desire as part of the install process (binary or source).

Skip



From mmclay@comcast.net  Mon Dec 30 06:54:40 2002
From: mmclay@comcast.net (Michael McLay)
Date: Mon, 30 Dec 2002 01:54:40 -0500
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <15887.58265.41467.93079@montanaro.dyndns.org>
References: <3E0E40AC.6040906@v.loewis.de>
 <008201c2afc8$c0659d00$125ffea9@oemcomputer>
 <15887.58265.41467.93079@montanaro.dyndns.org>
Message-ID: <200212300154.40233.mmclay@comcast.net>

On Monday 30 December 2002 01:11 am, Skip Montanaro wrote:
>     >> A requirement could be added to the PEP 2 that would require a
>     >> copyright disclosure be included in the package/module so that the
>     >> name "copyright" would retrieve the relevant copyright for that
>     >> package/module. A copyright that did not require disclosure would
>     >> return None.
>
>     Raymond> Though it makes the copyright accessible in a binary
>     Raymond> distribution, I don't think it fulfills the license
>     Raymond> requirements of being in printed in the documentation.
>
> Ah, but now you have a "machine" that will generate the bit of
> documentation you desire as part of the install process (binary or source).

+1

A script to generate the section in the documentation would keep the binary 
distribution in sync with the paper docs.  



From martin@v.loewis.de  Mon Dec 30 07:15:31 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 30 Dec 2002 08:15:31 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <ufzsgi9ao.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
Message-ID: <3E0FF293.80103@v.loewis.de>

David Abrahams wrote:
> Python extension module B calls shared library Q and uses Q's callback
> interface.  Because some of the callbacks need to use the Python API,
> and *might* be invoked by threads, they must all acquire the GIL.

Wrong. If the code in B that calls Q does not allow threads, the 
callbacks don't need to reacquire the GIL.

> Problem: using B while A is loaded breaks A: because B has installed
> callbacks in Q that acquire the GIL, A must also release the GIL
> before calling into Q.

Can you please explain what a callback is? Can the callbacks occur in 
the context of a different thread, i.e. different from the one that has 
installed the callback?

If it is a true callback (i.e. Q will call B back while being called 
from B), then this won't interfere at all with A.

Regards,
Martin



From martin@v.loewis.de  Mon Dec 30 07:20:33 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 30 Dec 2002 08:20:33 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <15887.58265.41467.93079@montanaro.dyndns.org>
References: <3E0E40AC.6040906@v.loewis.de>        <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>        <15887.33755.211400.175809@montanaro.dyndns.org>        <200212292349.29240.mclay@nist.gov>        <008201c2afc8$c0659d00$125ffea9@oemcomputer> <15887.58265.41467.93079@montanaro.dyndns.org>
Message-ID: <3E0FF3C1.1020307@v.loewis.de>

Skip Montanaro wrote:
> Ah, but now you have a "machine" that will generate the bit of documentation
> you desire as part of the install process (binary or source).

The question still is what is easiest for people producing Python 
binaries. Do they want to paste files from the CVS to fulfill the 
requirements, or would they prefer to print a variable in Python?

I agree with Raymond that collecting the statements alone in copyright 
won't help, since it does not meet the licensing criteria of some of the 
licenses.

Regards,
Martin



From martin@v.loewis.de  Mon Dec 30 07:24:19 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 30 Dec 2002 08:24:19 +0100
Subject: [Python-Dev] Proposed contribution to the library: PAX (another
 XML library)
In-Reply-To: <20021230000909.GC15340@laranja.org>
References: <20021230000909.GC15340@laranja.org>
Message-ID: <3E0FF4A3.7040602@v.loewis.de>

Lalo Martins wrote:
> Should I write a PEP?  Or a patch (silly, it would be a quite big patch)?

I think a library PEP would be needed. However, I would encourage you 
first to collect a user community for this code, and consider inclusion 
in the Python core only after a few years. One vehicle for potentially 
wider distribution of your code would be inclusion into PyXML, which I 
would help to perform if desired.

Regards,
Martin




From fredrik@pythonware.com  Mon Dec 30 10:25:27 2002
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 30 Dec 2002 11:25:27 +0100
Subject: [Python-Dev] Proposed contribution to the library: PAX (another XML library)
References: <20021230000909.GC15340@laranja.org>
Message-ID: <006f01c2afed$c6f15590$0900a8c0@spiff>

Lalo Martins wrote:

> PAX - the "Pythonic API for XML" - is a XML-handling library.
>=20
> Like with DOM, you're supposed to first parse the whole stream into =
the
> internal representation, and then operate on it.
>=20
> What distinguishes PAX is that its implementation is python-oriented; =
you
> iterate trough child nodes via the normal ways (for, getitem, and in =
2.2+,
> an iterator), the attributes are dictionary-based, etc.

how is this different from ElementTrees or all the other light-weight
DOM variants out there?

how much memory does it use, compared to minidom and the pyxml
tools?

how fast is it (compared to the same tools?  compared to more
optimized tools such as libxml and henry spencer's stuff?)

(I think the right approach would be to add it to the xml-sig =
distribution
first, and move it into the core only when you have more than a dozen
users...)

</F>



From mwh@python.net  Mon Dec 30 10:58:01 2002
From: mwh@python.net (Michael Hudson)
Date: 30 Dec 2002 10:58:01 +0000
Subject: [Python-Dev] GC at exit?
In-Reply-To: Aahz's message of "Sun, 29 Dec 2002 22:45:52 -0500"
References: <20021230034551.GA18622@panix.com>
Message-ID: <2m3cof3hae.fsf@starship.python.net>

Aahz <aahz@pythoncraft.com> writes:

> [Originally posted to comp.lang.python with no response; asking here
> before filing a bug report]
> 
> Is garbage collection supposed to run when Python exits?  The following
> program does not print any output, unless I uncomment the gc.collect()
> (or add a for loop that forces GC after creating the cycle):
> 
> import gc
> 
> class A:
>     pass
> 
> class B:
>     def __del__(self):
>         print "Bye-bye!"
> 
> a = A()
> b = A()
> a.b = b
> b.a = a
> a.x = B()
> 
> del a
> del b
> 
> #gc.collect()

One possibility is that the __del__ method *is* being run, but after
sys.stdout & so on has been torn down.  python -v can help, maybe?

Cheers,
M.

-- 
  Unfortunately, nigh the whole world is now duped into thinking that 
  silly fill-in forms on web pages is the way to do user interfaces.  
                                        -- Erik Naggum, comp.lang.lisp


From mwh@python.net  Mon Dec 30 12:41:28 2002
From: mwh@python.net (Michael Hudson)
Date: 30 Dec 2002 12:41:28 +0000
Subject: [Python-Dev] Fix for bug 597919 (compiler package vs SET_LINENO)
Message-ID: <2m1y3zd6h3.fsf@starship.python.net>

I've finally come up with a patch I'm 90% happy with for this problem,
got my iBook's modem talking to my ISP right and created a diff.

Unfortunately, SF's tracker isn't behaving and I'm about to disappear
from the internet for a day or so and 2.3a1 is due tomorrow, so it's
here:

   http://starship.python.net/crew/mwh/hacks/compiler-fixes-3.diff

There's one outstanding problem with the compiler package to do with
code of the form (unary operator)constant which is shown up by
test_grammar, but that aside I can compile the test suite and the
standard library using the patched compiler package and it runs fine.

I'd really like to see this patch in 2.3a1.  I'll fix the -literal
thing for 2.3a2, but I bet noone notices it in the mean time :)

Cheers,
M.

-- 
  at any rate, I'm satisfied that not only do they know which end of
  the pointy thing to hold, but where to poke it for maximum effect.
                                  -- Eric The Read, asr, on google.com


From guido@python.org  Mon Dec 30 13:05:10 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 30 Dec 2002 08:05:10 -0500
Subject: [Python-Dev] test_pyclbr failling
In-Reply-To: Your message of "Mon, 30 Dec 2002 00:04:36 EST."
 <15887.54244.866974.808863@slothrop.zope.com>
References: <20021230030406.GB12063@epoch.metaslash.com>
 <15887.54244.866974.808863@slothrop.zope.com>
Message-ID: <200212301305.gBUD5Ar14517@pcp02138704pcs.reston01.va.comcast.net>

> It would be helpful if the pyclbr tests could be re-written so that
> they don't depend on an arbitrary set of unrelated modules.  While
> people should run the full test suite, it seems common to run a
> limited set of tests when you're changing a single module and
> test_pyclbr always gets skipped.

I disagree.  I maintain pyclbr.py (having mostly rewritten its main
loop last August and another major update more recently) and usually a
failure is now a symptom of a bug in pyclbr.

> (I also wonder how useful this pyclbr module is for modern Python,
> since almost every module in test_pyclbr that has been modified in the
> last year has caused tests to fail.)

That was true before my most recent rewrite.  After it, I was able to
add back a bunch of tests that were commented out, and vastly reduce
the number of exceptions for other tests.  It's still not perfect, but
a lot more so.

pyclbr is used by IDLE's class and path browser commands, so it can't
be deprecated.

I'll look into the most recent random failure.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From dave@boost-consulting.com  Mon Dec 30 13:47:36 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 08:47:36 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <3E0FF293.80103@v.loewis.de> ("Martin v.
 =?iso-8859-1?q?L=F6wis"'s?= message of "Mon, 30 Dec 2002 08:15:31 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com> <3E0FF293.80103@v.loewis.de>
Message-ID: <uof73hb47.fsf@boost-consulting.com>

"Martin v. L=F6wis" <martin@v.loewis.de> writes:

> David Abrahams wrote:
>> Python extension module B calls shared library Q and uses Q's callback
>> interface.  Because some of the callbacks need to use the Python API,
>> and *might* be invoked by threads, they must all acquire the GIL.
>
> Wrong. If the code in B that calls Q does not allow threads, the
> callbacks don't need to reacquire the GIL.

I think you must be misunderstanding me.  These callbacks might be
invoked on threads that are not the Python main thread.=20=20

http://www.python.org/doc/current/api/threads.html says:

   "the rule exists that only the thread that has acquired the global
   interpreter lock may operate on Python objects or call Python/C API
   functions"

I am taking that statement at face value and assuming it means what it
says.  Note that these threads were not started by Python's thread API.

>> Problem: using B while A is loaded breaks A: because B has installed
>> callbacks in Q that acquire the GIL, A must also release the GIL
>> before calling into Q.
>
> Can you please explain what a callback is?=20

I'm trying to leave out irrelevant details, but a callback happens to
be a virtual function in a C++ class instance in this case.  These
callbacks implement behaviors of base classes supplied by the library,
Qt.  For the purposes of this discussion, it might just as well be a
'C' language pointer-to-function, though:

           void (*)()

> Can the callbacks occur in
> the context of a different thread, i.e. different from the one that
> has installed the callback?

I think the answer is yes, but I'm trying to leave out the irrelevant,
and I'm pretty sure that it doesn't matter one whit which thread
installs the callback.  What matters, AFAICT, is that the callback
might be invoked on a thread that's not Python's main thread, thus
must acquire the GIL, so if Python's main thread wants to call
something in Q that might invoke one of these callbacks, it must
release the GIL.

> If it is a true callback (i.e. Q will call B back while being called
> from B), then this won't interfere at all with A.

Maybe we have different ideas of what "callback" means.  In your
terms, it is not a "true callback".

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From lalo@laranja.org  Mon Dec 30 13:57:18 2002
From: lalo@laranja.org (Lalo Martins)
Date: Mon, 30 Dec 2002 11:57:18 -0200
Subject: [Python-Dev] Proposed contribution to the library: PAX (another XML library)
In-Reply-To: <006f01c2afed$c6f15590$0900a8c0@spiff>
References: <20021230000909.GC15340@laranja.org> <006f01c2afed$c6f15590$0900a8c0@spiff>
Message-ID: <20021230135718.GF15340@laranja.org>

On Mon, Dec 30, 2002 at 11:25:27AM +0100, Fredrik Lundh wrote:
> (I think the right approach would be to add it to the xml-sig distribution
> first, and move it into the core only when you have more than a dozen
> users...)

This seems reasonable.  Perhaps I was over-hasted by the threat of 2.3a ;-)
I will subscribe to the XML-sig asap.

[]s,
                                               |alo
                                               +----
--
            Those who trade freedom for security
               lose both and deserve neither.
--
http://www.laranja.org/                mailto:lalo@laranja.org
         pgp key: http://www.laranja.org/pessoal/pgp

Eu jogo RPG! (I play RPG)         http://www.eujogorpg.com.br/
GNU: never give up freedom                 http://www.gnu.org/


From skip@pobox.com  Mon Dec 30 14:32:25 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 30 Dec 2002 08:32:25 -0600
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <3E0FF3C1.1020307@v.loewis.de>
References: <3E0E40AC.6040906@v.loewis.de>
 <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>
 <15887.33755.211400.175809@montanaro.dyndns.org>
 <200212292349.29240.mclay@nist.gov>
 <008201c2afc8$c0659d00$125ffea9@oemcomputer>
 <15887.58265.41467.93079@montanaro.dyndns.org>
 <3E0FF3C1.1020307@v.loewis.de>
Message-ID: <15888.22777.811400.882470@montanaro.dyndns.org>

>>>>> "Martin" =3D=3D Martin v L=F6wis <martin@v.loewis.de> writes:

    Martin> Skip Montanaro wrote:
    >> Ah, but now you have a "machine" that will generate the bit of d=
ocumentation
    >> you desire as part of the install process (binary or source).

    Martin> The question still is what is easiest for people producing
    Martin> Python binaries. Do they want to paste files from the CVS t=
o
    Martin> fulfill the requirements, or would they prefer to print a
    Martin> variable in Python?

You simply have an automated post-install or binary distro build step w=
hich
does the little find-the-copyrights dance and dumps the result to somet=
hing
like LICENSES.txt or LICENSES.tex or LICENSES.hlp in the appropriate
directory and either declare it to be documentation or incorporate it i=
nto
the documentation delivered with the distribution.

People distributing Python with another package will have to do this so=
rt of
thing themselves, but that's not our concern.

Skip



From theller@python.net  Mon Dec 30 15:38:08 2002
From: theller@python.net (Thomas Heller)
Date: 30 Dec 2002 16:38:08 +0100
Subject: [Python-Dev] 'unhashable type' and new style classes
Message-ID: <3cofplen.fsf@python.net>

This code

  class X:
      def __cmp__(self, other):
          return 1

  c1 = X()

  d = {}
  d[c1] = None

raises a TypeError, unhashable type, since the class does
not define a __hash__ method.

This error is no longer raised when X derives from object.
Bug (or feature)?

Thomas



From aahz@pythoncraft.com  Mon Dec 30 14:59:12 2002
From: aahz@pythoncraft.com (Aahz)
Date: Mon, 30 Dec 2002 09:59:12 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <ufzsgi9ao.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
Message-ID: <20021230145912.GA8481@panix.com>

On Sun, Dec 29, 2002, David Abrahams wrote:
>
> Python extension module B calls shared library Q and uses Q's callback
> interface.  Because some of the callbacks need to use the Python API,
> and *might* be invoked by threads, they must all acquire the GIL.
> Because they also might be invoked by direct calls into Q, B must
> always release the GIL before calling anything in Q.

So you're saying that the callback functions in B acquire the GIL?

> Problem: using B while A is loaded breaks A: because B has installed
> callbacks in Q that acquire the GIL, A must also release the GIL
> before calling into Q.

Why?  The callbacks in B will simply hang until they acquire the GIL.

I think this thread might be better handled on c.l.py, at least until
it's understood well enough to be clear whether something does need to
change in Python.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"There are three kinds of lies: Lies, Damn Lies, and Statistics."  --Disraeli


From dave@boost-consulting.com  Mon Dec 30 15:15:22 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 10:15:22 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <20021230145912.GA8481@panix.com> (Aahz's message of "Mon, 30
 Dec 2002 09:59:12 -0500")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com>
Message-ID: <ud6njfshh.fsf@boost-consulting.com>

Aahz <aahz@pythoncraft.com> writes:

> On Sun, Dec 29, 2002, David Abrahams wrote:
>>
>> Python extension module B calls shared library Q and uses Q's callback
>> interface.  Because some of the callbacks need to use the Python API,
>> and *might* be invoked by threads, they must all acquire the GIL.
>> Because they also might be invoked by direct calls into Q, B must
>> always release the GIL before calling anything in Q.
>
> So you're saying that the callback functions in B acquire the GIL?

Yes.

>> Problem: using B while A is loaded breaks A: because B has installed
>> callbacks in Q that acquire the GIL, A must also release the GIL
>> before calling into Q.
>
> Why?  The callbacks in B will simply hang until they acquire the GIL.

If A doesn't release the GIL, one of its direct calls into Q from
Python may invoke a callback in B, which tries to acquire the lock
when it is already held.  This is a no-no.

I realize that the docs for PyEval_AcquireLock() say: "If this thread
already has the lock, a deadlock ensues", but the behavior we're
seeing is consistent with a scenario where trying to acquire an
already-held is a no-op and releasing it is unconditional.  Eventually
the GIL release in B's callback takes effect and when A returns to
Python there is no thread state.

> I think this thread might be better handled on c.l.py, at least
> until it's understood well enough to be clear whether something does
> need to change in Python.

I'm pretty certain I understand what's going on.  If it would be
better for you to take this to c.l.py, I'm happy to do so, but AFAICT
there _is_ a Python core issue here: there's no way to find out
whether you've already got the GIL**, so if you _might_ have to acquire
it, you must always acquire it.

**If I'm wrong about that, please let me know.  It isn't obvious from
  the documentation.

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From guido@python.org  Mon Dec 30 15:27:45 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 30 Dec 2002 10:27:45 -0500
Subject: [Python-Dev] 'unhashable type' and new style classes
In-Reply-To: Your message of "30 Dec 2002 16:38:08 +0100."
 <3cofplen.fsf@python.net>
References: <3cofplen.fsf@python.net>
Message-ID: <200212301527.gBUFRjT19772@odiug.zope.com>

> This code
> 
>   class X:
>       def __cmp__(self, other):
>           return 1
> 
>   c1 = X()
> 
>   d = {}
>   d[c1] = None
> 
> raises a TypeError, unhashable type, since the class does
> not define a __hash__ method.
> 
> This error is no longer raised when X derives from object.
> Bug (or feature)?

It's something in between. :-(

I've been struggling with this since 2.2 and not found a good
solution.  Built-in mutable types like list and dict deal with this by
having an explicit tp_hash slot that raises an exception.

Can you add a SF entry and assign it to me?  Any insight you might
have in the matter would be appreciated.

I don't think I'll get to this before the 2.3a1 release.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From jim@interet.com  Mon Dec 30 15:52:27 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 30 Dec 2002 10:52:27 -0500
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python,
 Not C
References: <200212241930.gBOJUYD10795@odiug.zope.com>              <3E0CBEE4.1020902@interet.com> <200212272215.gBRMFNw08691@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <3E106BBB.4020909@interet.com>

Guido van Rossum wrote:

>>Python already has an import hook, namely __import__.  PEP
>>302 adds three more hooks: sys.path_hooks, sys.meta_path and
>>sys.path_importer_hooks.  The idea of four import hooks is
>>already fishy.
> 
> There's no path_importer_hooks; there's path_importer_cache,

Ooops, sorry.  I meant sys.path_importer_cache.

> but
> that's not a new hook, it's a cache for the path_hooks hook.  So I
> think the PEP proposes only two new hooks.

I disagree.  It is publicly available in sys, and the PEP specifies
it can be replaced or modified.  And if you want to add a hook, it
is easier to put it into sys.path_importer_cache than to add it
to the list in sys.path_hooks.  If you add it to sys.path_hooks,
you must clear its entry in sys.path_importer_cache too.

>>That is a lot of external changes.  That is a lot of code
>>written in C.
> 
> Um, last I looked, most of the code written in C was specific to the
> zip importer; only a relatively small amount of code was added to
> import.c (about 10%).

I agree 90% of the C would be needed anyway.

>>I think the proper import hook design is to write Python's
>>import mechanism in Python along the lines of Greg's imputil.py
>>and Gordon's iu.py.

> That's a design that I have had in mind long ago, but I don't see it
> happening soon, because it would be a much larger overhaul of
> import.c.

Is there some reason you need hooks right away?  A current
application?

> I'm not so sure.  In practice, hooks are used for two things: import
> from other media than directories (e.g. zip files), and supporting
> additional filename extensions that trigger special transformations.

The PEP 302 hooks do not support the second case very well when the
files with special extensions are in zip archives.  Adding the hook
replaces the zip hook, as all PEP 302 hooks are alternatives to
each other.

JimA



From jim@interet.com  Mon Dec 30 16:16:31 2002
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 30 Dec 2002 11:16:31 -0500
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python,
 Not C
References: <r01050400-1022-3C57A01E19EB11D7A7C4003065D5E7E4@[10.0.0.2]>
Message-ID: <3E10715F.6040003@interet.com>

Just van Rossum wrote:

>>It changes the meaning of __import__.
> 
> Where did you get that idea?

I believe the base (unreplaced) __import__ function does
not find hooked imports.
It follows that __import__ will not find modules in zip
archives.

> This is backwards: iu.py implements a superset of PEP 302 (with
> different details). So you have that now.

Which is why I think we should fix iu.py instead of
add more public import hooks.  It is OK with me to
use your import hooks as an internal feature to
implement zip imports.

> PEP 302 allows customizing *parts* of the import mechanism
> without having to deal with most of these pitfalls and complications. It
> allows completely independend components to add hooks that will work
> together seamlessy. This is not true for replacing __import__.

I don't believe PEP 302 provides hooks that work together.  It
provides only one hook for each component of sys.path, and replaces
any hook that was already there.

JimA



From theller@python.net  Mon Dec 30 17:21:37 2002
From: theller@python.net (Thomas Heller)
Date: 30 Dec 2002 18:21:37 +0100
Subject: [Python-Dev] 'unhashable type' and new style classes
In-Reply-To: <200212301527.gBUFRjT19772@odiug.zope.com>
References: <3cofplen.fsf@python.net>
 <200212301527.gBUFRjT19772@odiug.zope.com>
Message-ID: <u1gvo21q.fsf@python.net>

Guido van Rossum <guido@python.org> writes:

> > This code
> > 
> >   class X:
> >       def __cmp__(self, other):
> >           return 1
> > 
> >   c1 = X()
> > 
> >   d = {}
> >   d[c1] = None
> > 
> > raises a TypeError, unhashable type, since the class does
> > not define a __hash__ method.
> > 
> > This error is no longer raised when X derives from object.
> > Bug (or feature)?
> 
> It's something in between. :-(
> 
> I've been struggling with this since 2.2 and not found a good
> solution.  Built-in mutable types like list and dict deal with this by
> having an explicit tp_hash slot that raises an exception.

Why doesn't the default implementation raise a TypeError (or be NULL),
and the immutable types override this?

> 
> Can you add a SF entry and assign it to me?  Any insight you might
> have in the matter would be appreciated.

Sure.

Thomas



From just@letterror.com  Mon Dec 30 16:29:52 2002
From: just@letterror.com (Just van Rossum)
Date: Mon, 30 Dec 2002 17:29:52 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <3E10715F.6040003@interet.com>
Message-ID: <r01050400-1022-EEDFFB721C1311D7A7C4003065D5E7E4@[10.0.0.3]>

James C. Ahlstrom wrote:

> >>It changes the meaning of __import__.
> > 
> > Where did you get that idea?
> 
> I believe the base (unreplaced) __import__ function does not find
> hooked imports. It follows that __import__ will not find modules in
> zip archives.

Not true. (It couldn't be true, simply because any import statement
*physically* goes through __import__.)

It _is_ true that zipimports won't neccesarily work when using existing
__import__ replacements. How big a deal that is, I honestly don't know.

> > This is backwards: iu.py implements a superset of PEP 302 (with
> > different details). So you have that now.
> 
> Which is why I think we should fix iu.py instead of
> add more public import hooks.  It is OK with me to
> use your import hooks as an internal feature to
> implement zip imports.

And I see *nothing* against making them public. They solve real problems
for real applications (besides zipimport) as well.

> > PEP 302 allows customizing *parts* of the import mechanism without
> > having to deal with most of these pitfalls and complications. It
> > allows completely independend components to add hooks that will
> > work together seamlessy. This is not true for replacing __import__.
> 
> I don't believe PEP 302 provides hooks that work together.  It
> provides only one hook for each component of sys.path, and replaces
> any hook that was already there.

And __import__ *does* provide this, how? Sure, PEP 302 is not a full
replacement of (say) ihooks.py, but it's nevertheless a vast improvement
over raw __import__ hooks.

Just


From guido@python.org  Mon Dec 30 16:30:10 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 30 Dec 2002 11:30:10 -0500
Subject: [Python-Dev] 'unhashable type' and new style classes
In-Reply-To: Your message of "30 Dec 2002 18:21:37 +0100."
 <u1gvo21q.fsf@python.net>
References: <3cofplen.fsf@python.net> <200212301527.gBUFRjT19772@odiug.zope.com>
 <u1gvo21q.fsf@python.net>
Message-ID: <200212301630.gBUGUAE03901@odiug.zope.com>

> > > This code
> > > 
> > >   class X:
> > >       def __cmp__(self, other):
> > >           return 1
> > > 
> > >   c1 = X()
> > > 
> > >   d = {}
> > >   d[c1] = None
> > > 
> > > raises a TypeError, unhashable type, since the class does
> > > not define a __hash__ method.
> > > 
> > > This error is no longer raised when X derives from object.
> > > Bug (or feature)?
> > 
> > It's something in between. :-(
> > 
> > I've been struggling with this since 2.2 and not found a good
> > solution.  Built-in mutable types like list and dict deal with this by
> > having an explicit tp_hash slot that raises an exception.
> 
> Why doesn't the default implementation raise a TypeError (or be NULL),
> and the immutable types override this?

The default implementation of __hash__ must match the default
implementation of __cmp__ (and rich comparisons, __eq__ etc.).  So the
default implementation cannot raise an exception, because objects are
defined to be immutable by default.  (Maybe this was a mistake, but
it's not so easy to change without causing backwards
incompatibilities.)

> > Can you add a SF entry and assign it to me?  Any insight you might
> > have in the matter would be appreciated.
> 
> Sure.

Thanks.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim@zope.com  Mon Dec 30 16:40:02 2002
From: tim@zope.com (Tim Peters)
Date: Mon, 30 Dec 2002 11:40:02 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <ud6njfshh.fsf@boost-consulting.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHKEJOEFAA.tim@zope.com>

[David Abrahams]
> ...
> but AFAICT there _is_ a Python core issue here: there's no way to find
> out whether you've already got the GIL**, so if you _might_ have to
acquire
> it, you must always acquire it.
>
> **If I'm wrong about that, please let me know.  It isn't obvious from
>   the documentation.

It's true -- you can't know whether you have the GIL, unless you code up
another layer of your own machinery to keep track of who has the GIL.  Mark
Hammond faces this issue (in all its multifacted glories) in the Win32
extensions, and built some C++ classes there to help him out.  It's
difficult at best, and last I looked (several years ago) I wasn't convinced
Mark's approach was 109% reliable.  The worst it gets in the Python core is
in posixmodule.c's _PyPclose, which needs to build a new interpreter state
from scratch, in order to build a new thread state from scratch, in order to
acquire the GIL, in order to call some Py API functions.  This isn't as hard
as it *can* get, because in that function we know the thread executing does
not hold the GIL.

The now-defunct Thread-SIG used to have bouts of angst over all this.  I
think we eventually figured out a better approach there, but it required
real work to implement, and nobody had time for that.  The only that's
changed since then is that nobody remembers the better approach anymore <0.9
wink>.

happy-new-year-ly y'rs  - tim



From martin@v.loewis.de  Mon Dec 30 17:01:31 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 30 Dec 2002 18:01:31 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <uof73hb47.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com> <3E0FF293.80103@v.loewis.de>
 <uof73hb47.fsf@boost-consulting.com>
Message-ID: <m3fzsfbfv8.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> > Wrong. If the code in B that calls Q does not allow threads, the
> > callbacks don't need to reacquire the GIL.
> 
> I think you must be misunderstanding me.  These callbacks might be
> invoked on threads that are not the Python main thread.  

Which thread is the main thread is not relevant at all.

What matters is whether the callback is invoked in the context of the
thread that installed it (i.e. as a result of calling a function in B).

>    "the rule exists that only the thread that has acquired the global
>    interpreter lock may operate on Python objects or call Python/C API
>    functions"
> 
> I am taking that statement at face value and assuming it means what it
> says.

And rightly so. Notice that it does not use the term "main thread".

> > Can you please explain what a callback is? 
> 
> I'm trying to leave out irrelevant details, but a callback happens to
> be a virtual function in a C++ class instance in this case.  These
> callbacks implement behaviors of base classes supplied by the library,
> Qt.  

Ok, now I know how a callback is installed (by redefining a virtual
method). The other question is: How is it invoked? I.e. who invokes
it, and why?

I suppose the immediate answer is "the library Q". However, that
library does not invoke it with out a trigger: What is that trigger?

> What matters, AFAICT, is that the callback might be invoked on a
> thread that's not Python's main thread, thus must acquire the GIL,
> so if Python's main thread wants to call something in Q that might
> invoke one of these callbacks, it must release the GIL.

Which thread is the main thread is completely irrelevant. If you have
something like

class MyB(B.Base):
  def overridden(self):
    print "This overrides a virtual function"

def doit():
  b = MyB()
  B.do_the_real_work(b) # will call Q, which will call the callback,
                        # which will call overridden

then there is no need for the C++ code in B to release the GIL, nor is
there a need for B to reacquire the GIL in the callback. This is
independent of whether doit is called in the main thread, or in the
context of any other thread: at the point where do_the_real_work is
called, the current thread holds the GIL. If it keeps the GIL, then
the GIL will be held at the point when the callback occurs, in the
context of the thread where the callback occurs.

> Maybe we have different ideas of what "callback" means.  In your
> terms, it is not a "true callback".

Then I'm still curious as to what triggers invocation of that virtual
function.

Regards,
Martin



From martin@v.loewis.de  Mon Dec 30 17:04:20 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 30 Dec 2002 18:04:20 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <ud6njfshh.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
Message-ID: <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> > So you're saying that the callback functions in B acquire the GIL?
> 
> Yes.

Ok. What would break if they wouldn't?

Regards,
Martin


From dave@boost-consulting.com  Mon Dec 30 17:26:34 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 12:26:34 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m3fzsfbfv8.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "30 Dec 2002 18:01:31 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com> <3E0FF293.80103@v.loewis.de>
 <uof73hb47.fsf@boost-consulting.com>
 <m3fzsfbfv8.fsf@mira.informatik.hu-berlin.de>
Message-ID: <ubs33ct9x.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> > Wrong. If the code in B that calls Q does not allow threads, the
>> > callbacks don't need to reacquire the GIL.
>>=20
>> I think you must be misunderstanding me.  These callbacks might be
>> invoked on threads that are not the Python main thread.=20=20
>
> Which thread is the main thread is not relevant at all.

Sorry, I got "main" confused with "current".  My point is that the
callback may be invoked by threads that don't hold the GIL.

> What matters is whether the callback is invoked in the context of
> the thread that installed it (i.e. as a result of calling a function
> in B).

I still don't see how the thread that installed it has any bearing.
Imagine there's a global function pointer variable somewhere.  Some
thread comes along and makes it point to some function f ("installs
the callback").  Now there are various ways that callback can be
called.  Some other thread may pick up the global variable at any time
and invoke the function it points to.  Why does it matter which thread
set the variable?

>> > Can you please explain what a callback is?=20
>>=20
>> I'm trying to leave out irrelevant details, but a callback happens to
>> be a virtual function in a C++ class instance in this case.  These
>> callbacks implement behaviors of base classes supplied by the library,
>> Qt.=20=20
>
> Ok, now I know how a callback is installed (by redefining a virtual
> method).=20

Technically, redefining a virtual function by itself doesn't do
anything.  You have to make an instance of the class which redefines
that function available to the library somehow.  But you knew that.

> The other question is: How is it invoked? I.e. who invokes
> it, and why?
>
> I suppose the immediate answer is "the library Q". However, that
> library does not invoke it with out a trigger: What is that trigger?

There are several ways, IIUC.  It may be invoked in response to direct
calls into Q's API (which may be made from a Python extension module).
It may also be invoked by some thread that Q has launched.

>> What matters, AFAICT, is that the callback might be invoked on a
>> thread that's not Python's main thread, thus must acquire the GIL,
>> so if Python's main thread wants to call something in Q that might
>> invoke one of these callbacks, it must release the GIL.
>
> Which thread is the main thread is completely irrelevant. If you have
> something like
>
> class MyB(B.Base):
>   def overridden(self):
>     print "This overrides a virtual function"
>
> def doit():
>   b =3D MyB()
>   B.do_the_real_work(b) # will call Q, which will call the callback,
>                         # which will call overridden
>
> then there is no need for the C++ code in B to release the GIL, nor
> is there a need for B to reacquire the GIL in the callback.  This is
> independent of whether doit is called in the main thread, or in the
> context of any other thread: at the point where do_the_real_work is
> called, the current thread holds the GIL.

Yes, we understand that problem.  I had this exact discussion with the
designer of B.  He explained that the problem is that Q might also
invoke the virtual function on a thread that is not holding the GIL.

> If it keeps the GIL, then the GIL will be held at the point when the
> callback occurs, in the context of the thread where the callback
> occurs.

Yes.

>> Maybe we have different ideas of what "callback" means.  In your
>> terms, it is not a "true callback".
>
> Then I'm still curious as to what triggers invocation of that virtual
> function.

Q, either directly via its API, or in some thread it started.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From dave@boost-consulting.com  Mon Dec 30 17:27:25 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 12:27:25 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "30 Dec 2002 18:04:20 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
Message-ID: <u7kdrct8i.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> > So you're saying that the callback functions in B acquire the GIL?
>>=20
>> Yes.
>
> Ok. What would break if they wouldn't?

All of the places where Q invokes the callback on arbitrary threads it
has started, but which don't hold the GIL.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From dave@boost-consulting.com  Mon Dec 30 17:29:01 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 12:29:01 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <20021230145912.GA8481@panix.com> (Aahz's message of "Mon, 30
 Dec 2002 09:59:12 -0500")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com>
Message-ID: <uznqnbele.fsf@boost-consulting.com>

Aahz <aahz@pythoncraft.com> writes:

> I think this thread might be better handled on c.l.py, at least until
> it's understood well enough to be clear whether something does need to
> change in Python.

BTW, I tried to comply with your request.  Currently the crossover
posting which I Bcc'd to python-dev is being held for moderator
approval:

python-dev-admin@python.org writes:

> Your mail to 'Python-Dev' with the subject
>
>     Extension modules, Threading, and the GIL
>
> Is being held until the list moderator can review it for approval.
>
> The reason it is being held:
>
>     Message has implicit destination
>
> Either the message will get posted to the list, or you will receive
> notification of the moderator's decision.
>

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From martin@v.loewis.de  Mon Dec 30 17:58:06 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 30 Dec 2002 18:58:06 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <u7kdrct8i.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
Message-ID: <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> > Ok. What would break if they wouldn't?
> 
> All of the places where Q invokes the callback on arbitrary threads it
> has started, but which don't hold the GIL.

So Q creates new threads which perform callbacks? But Q also performs
the callbacks when invoked from A? Sounds like a bug in Q to me...

Regards,
Martin



From martin@v.loewis.de  Mon Dec 30 18:11:02 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 30 Dec 2002 19:11:02 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <ubs33ct9x.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com> <3E0FF293.80103@v.loewis.de>
 <uof73hb47.fsf@boost-consulting.com>
 <m3fzsfbfv8.fsf@mira.informatik.hu-berlin.de>
 <ubs33ct9x.fsf@boost-consulting.com>
Message-ID: <m3fzsf9y2x.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> > What matters is whether the callback is invoked in the context of
> > the thread that installed it (i.e. as a result of calling a function
> > in B).
> 
> I still don't see how the thread that installed it has any bearing.

If the callbacks would only occur while Q is being called from B, and
in the context of the thread which also hosts that call, then B should
not release the lock. Then the callback would not need to reacquire
it.

> There are several ways, IIUC.  It may be invoked in response to direct
> calls into Q's API (which may be made from a Python extension module).
> It may also be invoked by some thread that Q has launched.

I find this surprising. Is this *any* API of Q that can trigger the
callbacks, or a dispatch_one_event call that can do so?

> Yes, we understand that problem.  I had this exact discussion with the
> designer of B.  He explained that the problem is that Q might also
> invoke the virtual function on a thread that is not holding the GIL.

I would more deeply question such a statement. It sounds like this
library uses the callback for event processing of some kind, in which
case the callbacks are only invoked if events are processed. It should
be possible to describe more precisely under what specific conditions
event processing occurs.

> > Then I'm still curious as to what triggers invocation of that virtual
> > function.
> 
> Q, either directly via its API, or in some thread it started.

Then calling this API must release the GIL.

Regards,
Martin


From dave@boost-consulting.com  Mon Dec 30 18:14:16 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 13:14:16 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "30 Dec 2002 18:58:06 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
Message-ID: <uptrjbchz.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> > Ok. What would break if they wouldn't?
>>=20
>> All of the places where Q invokes the callback on arbitrary threads it
>> has started, but which don't hold the GIL.
>
> So Q creates new threads which perform callbacks? But Q also performs
> the callbacks when invoked from A? Sounds like a bug in Q to me...

Why do you say that?  Q doesn't know anything about Python or its
constraints.  Why should it be prohibited from invoking these
callbacks in whatever way it deems appropriate for its problem domain?

I know, I know, all library authors should design with Python in mind
<wink>, but seriously, Q =3D=3D Qt, a library that's used extensively and
successfully by thousands.  It's conceivable that this is a serious
design flaw in Qt, but I'm inclined to disbelieve that.

I think rather that this is a library design which doesn't
interoperate well with Python's constraints on GIL manipulation.  In
this case, as Python is intended to be good for general
interoperability, it seems like Python ought to budge if possible.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From mal@lemburg.com  Mon Dec 30 18:14:10 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Mon, 30 Dec 2002 19:14:10 +0100
Subject: [Python-Dev] Mesenne Twister copyright notice
In-Reply-To: <3E0F6F98.1000404@v.loewis.de>
References: <F7288690-1B75-11D7-AC45-003065517236@oratrix.com> <3E0F6F98.1000404@v.loewis.de>
Message-ID: <3E108CF2.6010206@lemburg.com>

Martin v. L=F6wis wrote:
> Jack Jansen wrote:
>=20
>> On the other thread: will including the copyright=20
>> notices/acknowledgements in the documentation be good enough if we=20
>> don't always ship the documentation?=20
>=20
>=20
> This is exactly my concern. It is essentially for people producing=20
> binary distributions to decide how they want this to be organized.
>=20
>> Or should we put this in a special section that is always shipped,=20
>> even with binary installers that don't include documentation?
>=20
>=20
> For the specific case in question, I think MAL's suggestion of adding=20
> yet another file is sufficient - maintainers of binary distributions=20
> then need to collect all those text files and distribute them. Complyin=
g=20
>  with the various licenses is really something that the packagers most=20
> solve for themselves - the source distribution can only prepare things=20
> in a way that makes complying simple.

In a later mail I suggested adding these licenses to the standard
Python license file. This should probably be done as new section
explaining to which code the license applies and where the code
was obtained from.

Then binary distributions would only have to include this one
file (which they have to include anyway, since the Python license
requires this).

--=20
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From martin@v.loewis.de  Mon Dec 30 18:30:39 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 30 Dec 2002 19:30:39 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <uptrjbchz.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
Message-ID: <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> > So Q creates new threads which perform callbacks? But Q also performs
> > the callbacks when invoked from A? Sounds like a bug in Q to me...
> 
> Why do you say that?  Q doesn't know anything about Python or its
> constraints.  Why should it be prohibited from invoking these
> callbacks in whatever way it deems appropriate for its problem domain?

It is moderately evil for a library to create threads "under the
hoods", IMO; in some domains, that might be a reasonable thing to do,
provided there is a way for the application author to manage the
threads on a higher level (e.g. by limiting the total number of
threads that the library can create simultaneously).

If a library is creating new threads and invokes application code in
these threads, threading should follow a threading model. That
threading model has to be described, so that every application author
can rely on certain features. The threading model is part of the
library interface, just like the API.

It appears that Q has no threading model. That is truly evil.

In some cases, combining libraries with different threading models
just won't work. For example, I recently found that Tcl's appartment
threading model isn't really compatible with Python's GIL. It is
possible to achieve interworking to some respect, but there are
limitations that just can't be overcome (e.g. you just cannot invoke
event dispatching in a thread that hasn't originally create the Tcl
interpreter).

If the threading model of Q is unknown or undefined, you cannot expect
any kind of interworking.

> I think rather that this is a library design which doesn't
> interoperate well with Python's constraints on GIL manipulation.

It seems to me that there is no design in the library, and this is the
cause for the interoperability problem (or, perhaps, you just haven't
presented the design).

> In this case, as Python is intended to be good for general
> interoperability, it seems like Python ought to budge if possible.

As Tim explains, this might not be possible.

Regards,
Martin


From dave@boost-consulting.com  Mon Dec 30 18:49:04 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 13:49:04 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m37kdr9x68.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "30 Dec 2002 19:30:39 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
 <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>
Message-ID: <uadinbavz.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> > So Q creates new threads which perform callbacks? But Q also performs
>> > the callbacks when invoked from A? Sounds like a bug in Q to me...
>>=20
>> Why do you say that?  Q doesn't know anything about Python or its
>> constraints.  Why should it be prohibited from invoking these
>> callbacks in whatever way it deems appropriate for its problem
>> domain?
>
> It is moderately evil for a library to create threads "under the
> hoods", IMO; in some domains, that might be a reasonable thing to
> do, provided there is a way for the application author to manage the
> threads on a higher level (e.g. by limiting the total number of
> threads that the library can create simultaneously).

I am not intimately familiar with Qt; these threads may not in fact be
created "under the hood".  Whether they are or not is IMO irrelevant
to the problem we're having, because it's not how the thread is
started that matters.

> If a library is creating new threads and invokes application code in
> these threads, threading should follow a threading model. That
> threading model has to be described, so that every application
> author can rely on certain features. The threading model is part of
> the library interface, just like the API.
>
> It appears that Q has no threading model. That is truly evil.

Though I am not intimately familiar with Qt, I can assure you that it
*does* have a threading model.

> In some cases, combining libraries with different threading models
> just won't work. For example, I recently found that Tcl's appartment
> threading model isn't really compatible with Python's GIL. It is
> possible to achieve interworking to some respect, but there are
> limitations that just can't be overcome (e.g. you just cannot invoke
> event dispatching in a thread that hasn't originally create the Tcl
> interpreter).
>
> If the threading model of Q is unknown or undefined, you cannot expect
> any kind of interworking.
>
>> I think rather that this is a library design which doesn't
>> interoperate well with Python's constraints on GIL manipulation.
>
> It seems to me that there is no design in the library

I think that judgement is at best premature.

> and this is the cause for the interoperability problem (or,
> perhaps, you just haven't presented the design).

No, I haven't.  I'm not very familiar with it myself.  I'm just
relating information I've got from the author of PyQt, who is very
familiar with it.

>> In this case, as Python is intended to be good for general
>> interoperability, it seems like Python ought to budge if possible.
>
> As Tim explains, this might not be possible.

It sounded to me like Tim explained that it is possible but
unimplemented.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From dave@boost-consulting.com  Mon Dec 30 18:54:33 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 13:54:33 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m3fzsf9y2x.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "30 Dec 2002 19:11:02 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com> <3E0FF293.80103@v.loewis.de>
 <uof73hb47.fsf@boost-consulting.com>
 <m3fzsfbfv8.fsf@mira.informatik.hu-berlin.de>
 <ubs33ct9x.fsf@boost-consulting.com>
 <m3fzsf9y2x.fsf@mira.informatik.hu-berlin.de>
Message-ID: <u65tbbamu.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> > What matters is whether the callback is invoked in the context of
>> > the thread that installed it (i.e. as a result of calling a function
>> > in B).
>>=20
>> I still don't see how the thread that installed it has any bearing.
>
> If the callbacks would only occur while Q is being called from B,
> and in the context of the thread which also hosts that call, then B
> should not release the lock. Then the callback would not need to
> reacquire it.

AFAICT, that still has nothing to do with which thread installs the
callback.  It has everything to do with which threads may invoke the
callback, and whether they already hold the GIL.

>> There are several ways, IIUC.  It may be invoked in response to
>> direct calls into Q's API (which may be made from a Python
>> extension module).  It may also be invoked by some thread that Q
>> has launched.
>
> I find this surprising. Is this *any* API of Q that can trigger the
> callbacks, or a dispatch_one_event call that can do so?

I don't know; I'll have to defer to Phil for this answer.

>> Yes, we understand that problem.  I had this exact discussion with
>> the designer of B.  He explained that the problem is that Q might
>> also invoke the virtual function on a thread that is not holding
>> the GIL.
>
> I would more deeply question such a statement. It sounds like this
> library uses the callback for event processing of some kind, in
> which case the callbacks are only invoked if events are
> processed. It should be possible to describe more precisely under
> what specific conditions event processing occurs.

Probably.  I can't do it, though.

>> > Then I'm still curious as to what triggers invocation of that virtual
>> > function.
>>=20
>> Q, either directly via its API, or in some thread it started.
>
> Then calling this API must release the GIL.

Yes, that is clearly the case under present conditions.  The point is
that this is merely by virtue of the fact that some other extension
module may install a GIL-acquiring callback in Q.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From martin@v.loewis.de  Mon Dec 30 19:05:27 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 30 Dec 2002 20:05:27 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <uadinbavz.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
 <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>
 <uadinbavz.fsf@boost-consulting.com>
Message-ID: <m3y9678gzs.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> I am not intimately familiar with Qt

Ah, so Q is Qt :-) Care to reveal what A and B are, and which the
callback is that caused a deadlock?

> It sounded to me like Tim explained that it is possible but
> unimplemented.

Trying to channel Tim: All experts for this stuff have tried and
failed; Mark Hammond has a sort-of solution which Tim believes to be
strictly-speaking incorrect. While people may have thought they have a
solution, it is troubling that nobody can remember what the solution
is. Maybe the email signature just had not enough space to write it
down :-)

Regards,
Martin


From dave@boost-consulting.com  Mon Dec 30 19:19:55 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 14:19:55 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m3y9678gzs.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "30 Dec 2002 20:05:27 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
 <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>
 <uadinbavz.fsf@boost-consulting.com>
 <m3y9678gzs.fsf@mira.informatik.hu-berlin.de>
Message-ID: <uwulr9uw4.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> I am not intimately familiar with Qt
>
> Ah, so Q is Qt :-)=20

I actually revealed that several messages back.

> Care to reveal what A and B are

A is some extension module written by one of my users, that calls Qt
but doesn't install any callbacks.

B is some extension module which uses PyQt, thus installs callbacks
which invoke Python.

> and which the callback is that caused a deadlock?

There was no deadlock, as I've said.  The symptom is that Python
complains at some point that there's no thread state.  It goes away if
A releases the GIL before calling into Qt, and reacquires the GIL
afterwards.  I speculate that the callback releases the GIL when it is
finished, so that when A returns to Python there is no current thread.
By that time, the callback has completed, so it's hard to know for
sure which one it was.

>> It sounded to me like Tim explained that it is possible but
>> unimplemented.
>
> Trying to channel Tim: All experts for this stuff have tried and
> failed; Mark Hammond has a sort-of solution which Tim believes to be
> strictly-speaking incorrect. While people may have thought they have
> a solution, it is troubling that nobody can remember what the
> solution is. Maybe the email signature just had not enough space to
> write it down :-)

Hmm, it seems as though a mutex-protected record of which thread is
currently holding the GIL should be enough to handle it.  There must
be subtle details which complicate the problem, e.g. that there's no
portable/reliable way to identify the current thread (?)

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From guido@python.org  Mon Dec 30 21:12:00 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 30 Dec 2002 16:12:00 -0500
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: Your message of "Mon, 30 Dec 2002 17:29:52 +0100."
 <r01050400-1022-EEDFFB721C1311D7A7C4003065D5E7E4@[10.0.0.3]>
References: <r01050400-1022-EEDFFB721C1311D7A7C4003065D5E7E4@[10.0.0.3]>
Message-ID: <200212302112.gBULC0S04413@odiug.zope.com>

I should mention that I'm about to approve Just's additions to
import.c and very close to approving his zipimport.c, so you can
expect his checkins soon.  This means that sys.meta_path,
sys.path_hooks, and sys.path_importer_cache are new ways of hooking
import.  We can continue the discussion after 2.3a1 is released -- API
changes are okay during alpha testing.  I should note that a package's
__path__ continues to exist with semantics pretty close to what they
were; I am striving for Jython compatibility here.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From martin@v.loewis.de  Mon Dec 30 21:29:14 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 30 Dec 2002 22:29:14 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <uwulr9uw4.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
 <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>
 <uadinbavz.fsf@boost-consulting.com>
 <m3y9678gzs.fsf@mira.informatik.hu-berlin.de>
 <uwulr9uw4.fsf@boost-consulting.com>
Message-ID: <m3ptrj8ac5.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> There was no deadlock, as I've said.  

Yes, you only said this was a no-no, in

http://mail.python.org/pipermail/python-dev/2002-December/031449.html

I inferred from that (apparently incorrectly) that the no-no is a
deadlock.

> The symptom is that Python complains at some point that there's no
> thread state.  It goes away if A releases the GIL before calling
> into Qt, and reacquires the GIL afterwards.

Now I'm confused. In

http://mail.python.org/pipermail/python-dev/2002-December/031424.html

you said "A must also release the GIL" and then "the author of A may
have had no reason to believe anyone would install Python callbacks in
Q". From that I inferred that A does *not* release the GIL (as the
author had no reason to).

Now you are saying that A releases the GIL. Which one it is?

> I speculate that the callback releases the GIL when it is finished,
> so that when A returns to Python there is no current thread.

That would be a bug in the callback. If there was a thread state when
it was called, there should be a thread state when it returns.

> Hmm, it seems as though a mutex-protected record of which thread is
> currently holding the GIL should be enough to handle it. 

It depends on what "it" is, here. This one?

Q: Is there a way to find out whether the current thread holds the GIL?

If so, a mutex-protected record might work, but also might be
expensive.

Regards,
Martin


From Jack.Jansen@oratrix.com  Mon Dec 30 22:11:46 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Mon, 30 Dec 2002 23:11:46 +0100
Subject: [Python-Dev] 'unhashable type' and new style classes
In-Reply-To: <200212301630.gBUGUAE03901@odiug.zope.com>
Message-ID: <AF8E0C8A-1C43-11D7-B0FB-000A27B19B96@oratrix.com>

On maandag, dec 30, 2002, at 17:30 Europe/Amsterdam, Guido van Rossum 
wrote:
> The default implementation of __hash__ must match the default
> implementation of __cmp__ (and rich comparisons, __eq__ etc.).  So the
> default implementation cannot raise an exception, because objects are
> defined to be immutable by default.  (Maybe this was a mistake, but
> it's not so easy to change without causing backwards
> incompatibilities.)

Is it an option to enforce that these methods are overridden together, 
then? I.e. if you override any of them you must override all of them, 
otherwise you get a warning?
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From guido@python.org  Mon Dec 30 22:26:50 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 30 Dec 2002 17:26:50 -0500
Subject: [Python-Dev] 'unhashable type' and new style classes
In-Reply-To: Your message of "Mon, 30 Dec 2002 23:11:46 +0100."
 <AF8E0C8A-1C43-11D7-B0FB-000A27B19B96@oratrix.com>
References: <AF8E0C8A-1C43-11D7-B0FB-000A27B19B96@oratrix.com>
Message-ID: <200212302226.gBUMQon06368@odiug.zope.com>

> On maandag, dec 30, 2002, at 17:30 Europe/Amsterdam, Guido van Rossum 
> wrote:
> > The default implementation of __hash__ must match the default
> > implementation of __cmp__ (and rich comparisons, __eq__ etc.).  So the
> > default implementation cannot raise an exception, because objects are
> > defined to be immutable by default.  (Maybe this was a mistake, but
> > it's not so easy to change without causing backwards
> > incompatibilities.)
> 
> Is it an option to enforce that these methods are overridden together, 
> then? I.e. if you override any of them you must override all of them, 
> otherwise you get a warning?

That's nice, but I'm not sure I can see right away how to do the
warning.  Also, there are cases where a base class defines both
properly, and then a subclass overrides one but not the other.  Is
that worth a warning or not?  That depends on how they change it!

--Guido van Rossum (home page: http://www.python.org/~guido/)


From Jack.Jansen@oratrix.com  Mon Dec 30 23:15:03 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Tue, 31 Dec 2002 00:15:03 +0100
Subject: [Python-Dev] 'unhashable type' and new style classes
In-Reply-To: <200212302226.gBUMQon06368@odiug.zope.com>
Message-ID: <86977B19-1C4C-11D7-B0FB-000A27B19B96@oratrix.com>

On maandag, dec 30, 2002, at 23:26 Europe/Amsterdam, Guido van Rossum 
wrote:

>> On maandag, dec 30, 2002, at 17:30 Europe/Amsterdam, Guido van Rossum
>> wrote:
>>> The default implementation of __hash__ must match the default
>>> implementation of __cmp__ (and rich comparisons, __eq__ etc.).  So 
>>> the
>>> default implementation cannot raise an exception, because objects are
>>> defined to be immutable by default.  (Maybe this was a mistake, but
>>> it's not so easy to change without causing backwards
>>> incompatibilities.)
>>
>> Is it an option to enforce that these methods are overridden together,
>> then? I.e. if you override any of them you must override all of them,
>> otherwise you get a warning?
>
> That's nice, but I'm not sure I can see right away how to do the
> warning.

I assume there's some equivalent of PyType_Ready, or not? Wouldn't that 
be the
right spot?

> Also, there are cases where a base class defines both
> properly, and then a subclass overrides one but not the other.  Is
> that worth a warning or not?  That depends on how they change it!

It's easy enough to add dummy methods that just call the same method in 
the base
class. But still: this whole warning idea could be overkill, that's for 
you to decide:-)
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From neal@metaslash.com  Mon Dec 30 23:41:39 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Mon, 30 Dec 2002 18:41:39 -0500
Subject: [Python-Dev] bugs in stdlib
Message-ID: <20021230234139.GE12063@epoch.metaslash.com>

Here's some problems I found in the stdlib.
I already fixed one problem in ConfigParser, but I'm not sure it's correct.

Lib/cgitb.py:202: No global (path) found
Lib/cgitb.py:204: No global (path) found
Lib/ConfigParser.py:581: Invalid arguments to (__init__), got 1, expected 4
Lib/imaplib.py:444: No global (root) found
Lib/pprint.py:118: Invalid arguments to (format), got 3, expected 4
Lib/pprint.py:121: Invalid arguments to (format), got 3, expected 4

Neal


From DavidA@ActiveState.com  Tue Dec 31 00:31:38 2002
From: DavidA@ActiveState.com (David Ascher)
Date: Mon, 30 Dec 2002 16:31:38 -0800
Subject: [Python-Dev] Mesenne Twister copyright notice
References: <F7288690-1B75-11D7-AC45-003065517236@oratrix.com> <3E0F6F98.1000404@v.loewis.de> <3E108CF2.6010206@lemburg.com>
Message-ID: <3E10E56A.5020404@ActiveState.com>

M.-A. Lemburg wrote:

> In a later mail I suggested adding these licenses to the standard
> Python license file. This should probably be done as new section
> explaining to which code the license applies and where the code
> was obtained from.
>
> Then binary distributions would only have to include this one
> file (which they have to include anyway, since the Python license
> requires this).

This would suit us (ActiveState) just fine.  In general, we don't mind 
doing whatever is necessary to give credit where credit is due (as well 
as comply with the license in general, of course) -- but the simpler it 
is, the more likely it is to actually be carried through -- the more 
automatable the better as well.

--david



From amk@amk.ca  Tue Dec 31 01:19:33 2002
From: amk@amk.ca (A.M. Kuchling)
Date: Mon, 30 Dec 2002 20:19:33 -0500
Subject: [Python-Dev] Don't touch whatsnew23.tex
Message-ID: <200212310119.gBV1JXw16395@nyman.amk.ca>

I'm in the middle of updating the "What's New" document for 2.3a1.
Please don't make any checkins to it; if you want to make a change,
please e-mail me a patch instead.  (Should be finished later this
evening, unless someone adds more features first...)

--amk                                                             (www.amk.ca)
Thank you for letting me borrow your objects.
      -- Ute Lemper in concert, March 13, 1997



From aahz@pythoncraft.com  Tue Dec 31 00:57:36 2002
From: aahz@pythoncraft.com (Aahz)
Date: Mon, 30 Dec 2002 19:57:36 -0500
Subject: [Python-Dev] GC at exit?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEDFDFAB.tim.one@comcast.net>
References: <20021230034551.GA18622@panix.com> <LNBBLJKPBEHFEDALKOLCMEDFDFAB.tim.one@comcast.net>
Message-ID: <20021231005736.GA15066@panix.com>

On Sun, Dec 29, 2002, Tim Peters wrote:
> [Aahz]
>>
>> Is garbage collection supposed to run when Python exits?
> 
> The language reference manual doesn't promise that any garbage will be
> collected, ever.  So, no, "supposed to" doesn't apply.

Thanks.  I'll tell people to run gc.collect() at the end of their
applications if they care.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"There are three kinds of lies: Lies, Damn Lies, and Statistics."  --Disraeli


From esr@thyrsus.com  Tue Dec 31 01:43:55 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Mon, 30 Dec 2002 20:43:55 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <20021230234139.GE12063@epoch.metaslash.com>
References: <20021230234139.GE12063@epoch.metaslash.com>
Message-ID: <20021231014355.GA14992@thyrsus.com>

Neal Norwitz <neal@metaslash.com>: 
> Here's some problems I found in the stdlib.
> I already fixed one problem in ConfigParser, but I'm not sure it's correct.

I've got another fix and some enhancements in the works for
ConfigParser.  Passes all regression tests, but I'm field-testing
before I commit.  Just because I'm nervous about patch collisions,
what code did your patch affect?
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From dave@boost-consulting.com  Tue Dec 31 04:11:06 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 23:11:06 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m3ptrj8ac5.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "30 Dec 2002 22:29:14 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
 <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>
 <uadinbavz.fsf@boost-consulting.com>
 <m3y9678gzs.fsf@mira.informatik.hu-berlin.de>
 <uwulr9uw4.fsf@boost-consulting.com>
 <m3ptrj8ac5.fsf@mira.informatik.hu-berlin.de>
Message-ID: <ud6ni7rqd.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> There was no deadlock, as I've said.=20=20
>
> Yes, you only said this was a no-no, in
>
> http://mail.python.org/pipermail/python-dev/2002-December/031449.html
>
> I inferred from that (apparently incorrectly) that the no-no is a
> deadlock.

In that very message, I also wrote:

    I realize that the docs for PyEval_AcquireLock() say: "If this
    thread already has the lock, a deadlock ensues", but the behavior
    we're seeing is consistent with a scenario where trying to acquire
    an already-held is a no-op and releasing it is unconditional.
    Eventually the GIL release in B's callback takes effect and when A
    returns to Python there is no thread state.

>> The symptom is that Python complains at some point that there's no
>> thread state.  It goes away if A releases the GIL before calling
>> into Qt, and reacquires the GIL afterwards.
>
> Now I'm confused. In
>
> http://mail.python.org/pipermail/python-dev/2002-December/031424.html
>
> you said "A must also release the GIL"=20

Yes, that's the inevitable conclusion.

> and then "the author of A may have had no reason to believe anyone
> would install Python callbacks in Q". From that I inferred that A
> does *not* release the GIL (as the author had no reason to).

Yes, not releasing the GIL in A was fine until B came along and
installed callbacks in Q which acquire the GIL.
>
> Now you are saying that A releases the GIL. Which one it is?

No, I am not saying A releases the GIL.  I am saying that A that must
release the GIL if it is to work properly in the presence of B.  A is
currently broken in the presence of B.  The addition of B to the
system places a new constraint on A.

>> I speculate that the callback releases the GIL when it is finished,
>> so that when A returns to Python there is no current thread.
>
> That would be a bug in the callback.=20

Not if it has previously acquired the GIL.

> If there was a thread state when it was called, there should be a
> thread state when it returns.

Yes, the whole problem is that there's no way to know whether there's
a thread state.

>> Hmm, it seems as though a mutex-protected record of which thread is
>> currently holding the GIL should be enough to handle it.=20
>
> It depends on what "it" is, here. This one?
>
> Q: Is there a way to find out whether the current thread holds the GIL?
>
> If so, a mutex-protected record might work, but also might be
> expensive.

Yes.  I assume that acquiring the GIL already needs to do
synchronization, though.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From martin@v.loewis.de  Tue Dec 31 10:49:19 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 31 Dec 2002 11:49:19 +0100
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <ud6ni7rqd.fsf@boost-consulting.com>
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
 <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>
 <uadinbavz.fsf@boost-consulting.com>
 <m3y9678gzs.fsf@mira.informatik.hu-berlin.de>
 <uwulr9uw4.fsf@boost-consulting.com>
 <m3ptrj8ac5.fsf@mira.informatik.hu-berlin.de>
 <ud6ni7rqd.fsf@boost-consulting.com>
Message-ID: <m3fzseea4w.fsf@mira.informatik.hu-berlin.de>

David Abrahams <dave@boost-consulting.com> writes:

> >> The symptom is that Python complains at some point that there's no
> >> thread state.  It goes away if A releases the GIL before calling
> >> into Qt, and reacquires the GIL afterwards.
[...]
> No, I am not saying A releases the GIL.  

"...there is no thread state. It [the thread state] goes away if A
releases the GIL ..."

>From that I inferred that A releases the GIL, since you said that
there is no thread state. Rereading your message, I now see that you
meant "It [the problem] goes away".

So I now understand that you reported that there is no deadlock, and
that A does not release the GIL, and that Python reports that there is
no thread state "when A returns to Python". You also report that B
acquires the GIL.

I can't understand why this happens. How does B acquire the GIL?

Assuming that B uses PyEval_AcquireThread/PyEval_ReleaseThread, I
would expect that 
a) there is a deadlock if this happens in a context of a call to
   A, since the GIL is already held, and (if, for some reason,
   locks are recursive on this platform), 
b) the code

	if (PyThreadState_Swap(tstate) != NULL)
		Py_FatalError(
			"PyEval_AcquireThread: non-NULL old thread state");

   should trigger, as there is an old thread state.

So I infer that B does not use
PyEval_AcquireThread/PyEval_ReleaseThread. What else does it use?

> >> I speculate that the callback releases the GIL when it is finished,
> >> so that when A returns to Python there is no current thread.
> >
> > That would be a bug in the callback. 
> 
> Not if it has previously acquired the GIL.

Even then, but I refrain from speculating how B's code to release the
lock actually looks.

> > If there was a thread state when it was called, there should be a
> > thread state when it returns.
> 
> Yes, the whole problem is that there's no way to know whether there's
> a thread state.

Wrong. If B acquires the GIL, B must use some thread state to do
so. It must install that thread state through PyThreadState_Swap,
directly or indirectly. That will return the old thread state, or
NULL.

> > If so, a mutex-protected record might work, but also might be
> > expensive.
> 
> Yes.  I assume that acquiring the GIL already needs to do
> synchronization, though.

Sure. But with that proposed change, you have not only the GIL lock
call (which is a single sem_wait call on Posix, and an
InterlockedCompareExchange call on Win32). You also get a mutex call,
and a call to find out the current thread.

Regards,
Martin


From neal@metaslash.com  Tue Dec 31 12:19:03 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 31 Dec 2002 07:19:03 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <20021231014355.GA14992@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com>
Message-ID: <20021231121903.GK12063@epoch.metaslash.com>

On Mon, Dec 30, 2002 at 08:43:55PM -0500, Eric S. Raymond wrote:
> 
> I've got another fix and some enhancements in the works for
> ConfigParser.  Passes all regression tests, but I'm field-testing
> before I commit.  Just because I'm nervous about patch collisions,
> what code did your patch affect?

I added class InterpolationSyntaxError(Error).  Fred quickly fixed it
:-), added a bunch of docstrings, fixed the constructor call I noted,
and added some tests.

Neal


From dave@boost-consulting.com  Tue Dec 31 12:24:31 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Tue, 31 Dec 2002 07:24:31 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
In-Reply-To: <m3fzseea4w.fsf@mira.informatik.hu-berlin.de> (martin@v.loewis.de's
 message of "31 Dec 2002 11:49:19 +0100")
References: <ufzsgi9ao.fsf@boost-consulting.com>
 <20021230145912.GA8481@panix.com> <ud6njfshh.fsf@boost-consulting.com>
 <m3bs33bfqj.fsf@mira.informatik.hu-berlin.de>
 <u7kdrct8i.fsf@boost-consulting.com>
 <m3k7hr9yoh.fsf@mira.informatik.hu-berlin.de>
 <uptrjbchz.fsf@boost-consulting.com>
 <m37kdr9x68.fsf@mira.informatik.hu-berlin.de>
 <uadinbavz.fsf@boost-consulting.com>
 <m3y9678gzs.fsf@mira.informatik.hu-berlin.de>
 <uwulr9uw4.fsf@boost-consulting.com>
 <m3ptrj8ac5.fsf@mira.informatik.hu-berlin.de>
 <ud6ni7rqd.fsf@boost-consulting.com>
 <m3fzseea4w.fsf@mira.informatik.hu-berlin.de>
Message-ID: <usmwe2x6o.fsf@boost-consulting.com>

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> >> The symptom is that Python complains at some point that there's no
>> >> thread state.  It goes away if A releases the GIL before calling
>> >> into Qt, and reacquires the GIL afterwards.
> [...]
>> No, I am not saying A releases the GIL.=20=20
>
> "...there is no thread state. It [the thread state] goes away if A
> releases the GIL ..."
>
> From that I inferred that A releases the GIL, since you said that
> there is no thread state. Rereading your message, I now see that you
> meant "It [the problem] goes away".

Right.

> So I now understand that you reported that there is no deadlock, and
> that A does not release the GIL, and that Python reports that there is
> no thread state "when A returns to Python". You also report that B
> acquires the GIL.
>
> I can't understand why this happens. How does B acquire the GIL?
>
> Assuming that B uses PyEval_AcquireThread/PyEval_ReleaseThread, I
> would expect that=20
> a) there is a deadlock if this happens in a context of a call to
>    A, since the GIL is already held, and (if, for some reason,
>    locks are recursive on this platform),=20
> b) the code
>
> 	if (PyThreadState_Swap(tstate) !=3D NULL)
> 		Py_FatalError(
> 			"PyEval_AcquireThread: non-NULL old thread state");
>
>    should trigger, as there is an old thread state.
>
> So I infer that B does not use
> PyEval_AcquireThread/PyEval_ReleaseThread. What else does it use?


Looking at the SIP sources, it appears to be using
PyEval_SaveThread/PyEval_RestoreThread, but I'd have to ask Phil to
weigh in on this one to know for sure.

Here's a stack backtrace reported by my user.  You can ignore the
oddness of frame #4; the SIP author is patching Python's instance
method table, but has convinced me that what he's doing is harmless
(it's still evil, of course <wink>).

  Program received signal SIGSEGV, Segmentation fault.
  [Switching to Thread 1024 (LWP 5948)]
  PyErr_SetObject (exception=3D0x8108a8c, value=3D0x81ab208) at Python/erro=
rs.c:39
  (gdb) bt
  #0  PyErr_SetObject (exception=3D0x8108a8c, value=3D0x81ab208)
      at Python/errors.c:39
  #1  0x08087ac7 in PyErr_Format (exception=3D0x8108a8c,=20
      format=3D0x80df620 "%.50s instance has no attribute '%.400s'")
      at Python/errors.c:408
  #2  0x080b0467 in instance_getattr1 (inst=3D0x82c5654, name=3D0x8154558)
      at Objects/classobject.c:678
  #3  0x080b3e35 in instance_getattr (inst=3D0x82c5654, name=3D0x8154558)
      at Objects/classobject.c:715
  #4  0x40cd2a43 in instanceGetAttr ()
     from /usr/local/lib/python2.2/site-packages/libsip.so
  #5  0x08056794 in PyObject_GetAttr (v=3D0x82c5654, name=3D0x8154558)
      at Objects/object.c:1108
  #6  0x0807705e in eval_frame (f=3D0x811a974) at Python/ceval.c:1784
  #7  0x0807866e in PyEval_EvalCodeEx (co=3D0x8161de0, globals=3D0x81139b4,=
=20
      locals=3D0x81139b4, args=3D0x0, argcount=3D0, kws=3D0x0, kwcount=3D0,=
 defs=3D0x0,=20
      defcount=3D0, closure=3D0x0) at Python/ceval.c:2595
  #8  0x0807a700 in PyEval_EvalCode (co=3D0x8161de0, globals=3D0x81139b4,=20
      locals=3D0x81139b4) at Python/ceval.c:481
  #9  0x080950b1 in run_node (n=3D0x81263b8,=20
      filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p=
y",=20
      globals=3D0x81139b4, locals=3D0x81139b4, flags=3D0xbffffac4)
      at Python/pythonrun.c:1079
  #10 0x08095062 in run_err_node (n=3D0x81263b8,=20
      filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p=
y",=20
      globals=3D0x81139b4, locals=3D0x81139b4, flags=3D0xbffffac4)
      at Python/pythonrun.c:1066
  #11 0x08094ccb in PyRun_FileExFlags (fp=3D0x8104038,=20
      filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p=
y",=20
      start=3D257, globals=3D0x81139b4, locals=3D0x81139b4, closeit=3D1,=20
      flags=3D0xbffffac4) at Python/pythonrun.c:1057
  #12 0x080938b1 in PyRun_SimpleFileExFlags (fp=3D0x8104038,=20
      filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p=
y",=20
      closeit=3D1, flags=3D0xbffffac4) at Python/pythonrun.c:685
  #13 0x0809481f in PyRun_AnyFileExFlags (fp=3D0x8104038,=20
      filename=3D0xbffffca2 "/home/pfkeb/hippodraw-BUILD/testsuite/dclock.p=
y",=20
      closeit=3D1, flags=3D0xbffffac4) at Python/pythonrun.c:495
  #14 0x08053632 in Py_Main (argc=3D2, argv=3D0xbffffb54) at Modules/main.c=
:364
  #15 0x08052ee6 in main (argc=3D2, argv=3D0xbffffb54) at Modules/python.c:=
10
  #16 0x40088627 in __libc_start_main (main=3D0x8052ed0 <main>, argc=3D2,=20
      ubp_av=3D0xbffffb54, init=3D0x80522d4 <_init>, fini=3D0x80cf610 <_fin=
i>,=20
      rtld_fini=3D0x4000dcd4 <_dl_fini>, stack_end=3D0xbffffb4c)
      at ../sysdeps/generic/libc-start.c:129
  (gdb)=20

     On the line of the error

          oldtype =3D tstate->curexc_type;

  (gdb) p tstate
  $1 =3D (PyThreadState *) 0x0
  (gdb)=20

>> > If there was a thread state when it was called, there should be a
>> > thread state when it returns.
>>=20
>> Yes, the whole problem is that there's no way to know whether there's
>> a thread state.
>
> Wrong. If B acquires the GIL, B must use some thread state to do
> so. It must install that thread state through PyThreadState_Swap,
> directly or indirectly. That will return the old thread state, or
> NULL.

Let me rephrase: the whole problem is that there's no way to know if
you have the interpreter lock.  You can't call PyThreadState_Swap to
find out if there's a thread state if you don't have the interpreter
lock.  You can't acquire the lock if you already have it.

>> > If so, a mutex-protected record might work, but also might be
>> > expensive.
>>=20
>> Yes.  I assume that acquiring the GIL already needs to do
>> synchronization, though.
>
> Sure. But with that proposed change, you have not only the GIL lock
> call (which is a single sem_wait call on Posix, and an
> InterlockedCompareExchange call on Win32). You also get a mutex
> call, and a call to find out the current thread.

There you go, it's a harder problem than I thought ;-)

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution



From mal@lemburg.com  Tue Dec 31 13:08:19 2002
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 31 Dec 2002 14:08:19 +0100
Subject: [Python-Dev] Third-Party Licenses (Mesenne Twister copyright
 notice)
In-Reply-To: <3E10E56A.5020404@ActiveState.com>
References: <F7288690-1B75-11D7-AC45-003065517236@oratrix.com>	<3E0F6F98.1000404@v.loewis.de> <3E108CF2.6010206@lemburg.com> <3E10E56A.5020404@ActiveState.com>
Message-ID: <3E1196C3.70805@lemburg.com>

David Ascher wrote:
> M.-A. Lemburg wrote:
> 
>> In a later mail I suggested adding these licenses to the standard
>> Python license file. This should probably be done as new section
>> explaining to which code the license applies and where the code
>> was obtained from.
>>
>> Then binary distributions would only have to include this one
>> file (which they have to include anyway, since the Python license
>> requires this).
> 
> 
> This would suit us (ActiveState) just fine.  In general, we don't mind 
> doing whatever is necessary to give credit where credit is due (as well 
> as comply with the license in general, of course) -- but the simpler it 
> is, the more likely it is to actually be carried through -- the more 
> automatable the better as well.

So what's the general feeling ? Should we take this approach ?

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/



From guido@python.org  Tue Dec 31 14:02:33 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 09:02:33 -0500
Subject: [Python-Dev] Python 2.3a1 release status
Message-ID: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net>

I'm planning for a release by 6pm EST today.

I'm happy with the changes that've been checked in overnight and the
quality control checks that have been made.  I'm waiting for one more
big change to land: an important API change to datetime.c that was
already implemented in the .py version in the sandbox.  I expect this
around lunchtime (EST) today.  Once that change is committed, I'll
create a release branch, declare a checkin moratorium for people
outside PythonLabs, and start working on the final stretch of the
release process (see PEP 101 if you want to know how much work that
is).

If you know of other pending changes that haven't been committed,
either check them in (only if they're stable, tested and documented,
of course!) or let me know ASAP.

Jack Jansen: I don't expect that you're quite ready with the Mac tree.
Or are you?  Let me know what you want me to say about the Mac in the
announcement and on the web.

The draft text for the website is already visible at
http://www.python.org/2.3/.  If you think you can use this to get some
short-term gain on the stock market, please share. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Tue Dec 31 14:09:37 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 31 Dec 2002 15:09:37 +0100
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C
In-Reply-To: <3E106BBB.4020909@interet.com>
Message-ID: <r01050400-1022-8AC2815C1CC911D7A7C4003065D5E7E4@[10.0.0.3]>

[JvR]
> > but [sys.path_importer_cache is] not a new hook, it's a cache for
> > the path_hooks hook.  So I think the PEP proposes only two new
> > hooks.

[James C. Ahlstrom]
> I disagree.  It is publicly available in sys, and the PEP specifies
> it can be replaced or modified.  And if you want to add a hook, it
> is easier to put it into sys.path_importer_cache than to add it
> to the list in sys.path_hooks.  If you add it to sys.path_hooks,
> you must clear its entry in sys.path_importer_cache too.

This is all only half true. sys.path_importer_cache is exposed so you
can *clear* it in case you install a path hook that should take over
sys.path items that may already have a handler. To only partially clear
it is advanced usage. By no means you're supposed to *add* to
sys.path_importer_cache explicitly, as someone after you may clear it
again, and then you're hosed if there's no proper path hook.

I also disagree that adding an importer to sys.path_importer_cache would
always be *easier* than doing it the right way by adding a hook to
sys.path_hooks. If would be somewhat easier for "cookie" path items, but
I think in 99% of these cases a path-less importer on sys.meta_path
would work just as well.

I guess I should start on some proper documentation...

Just


From martin@v.loewis.de  Tue Dec 31 14:46:51 2002
From: martin@v.loewis.de (Martin v. =?iso-8859-15?q?L=F6wis?=)
Date: 31 Dec 2002 15:46:51 +0100
Subject: [Python-Dev] Python 2.3a1 release status
In-Reply-To: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net>
References: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <m3ptrickkk.fsf@mira.informatik.hu-berlin.de>

Guido van Rossum <guido@python.org> writes:

> I'm planning for a release by 6pm EST today.

It appears that IDLEfork will not be merged for 2.3a1. Is there a plan
to merge it for 2.3?

Regards,
Martin



From esr@thyrsus.com  Tue Dec 31 14:56:39 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 09:56:39 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <20021231121903.GK12063@epoch.metaslash.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com>
Message-ID: <20021231145639.GA22240@thyrsus.com>

Neal Norwitz <neal@metaslash.com>:
> On Mon, Dec 30, 2002 at 08:43:55PM -0500, Eric S. Raymond wrote:
> > I've got another fix and some enhancements in the works for
> > ConfigParser.  Passes all regression tests, but I'm field-testing
> > before I commit.  Just because I'm nervous about patch collisions,
> > what code did your patch affect?
> 
> I added class InterpolationSyntaxError(Error).  Fred quickly fixed it
> :-), added a bunch of docstrings, fixed the constructor call I noted,
> and added some tests.

Doesn't sound like there should be a problem. Most of my changes are to the
_read logic -- I also added a bunch of regression tests for the new features, 
but I did it in such a way that collision seems unlikely.  I'll check them 
in today or tomorrow and make sure everything is copacetic.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From guido@python.org  Tue Dec 31 15:18:31 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 10:18:31 -0500
Subject: [Python-Dev] Python 2.3a1 release status
In-Reply-To: Your message of "31 Dec 2002 15:46:51 +0100."
 <m3ptrickkk.fsf@mira.informatik.hu-berlin.de>
References: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net>
 <m3ptrickkk.fsf@mira.informatik.hu-berlin.de>
Message-ID: <200212311518.gBVFIV914092@odiug.zope.com>

> > I'm planning for a release by 6pm EST today.
> 
> It appears that IDLEfork will not be merged for 2.3a1. Is there a plan
> to merge it for 2.3?

Yes, I hope they'll be ready for 2.3a2.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Tue Dec 31 15:22:18 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 10:22:18 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: Your message of "Tue, 31 Dec 2002 09:56:39 EST."
 <20021231145639.GA22240@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
Message-ID: <200212311522.gBVFMIY14143@odiug.zope.com>

> Neal Norwitz <neal@metaslash.com>:
> > On Mon, Dec 30, 2002 at 08:43:55PM -0500, Eric S. Raymond wrote:
> > > I've got another fix and some enhancements in the works for
> > > ConfigParser.  Passes all regression tests, but I'm field-testing
> > > before I commit.  Just because I'm nervous about patch collisions,
> > > what code did your patch affect?
> > 
> > I added class InterpolationSyntaxError(Error).  Fred quickly fixed it
> > :-), added a bunch of docstrings, fixed the constructor call I noted,
> > and added some tests.
> 
> Doesn't sound like there should be a problem. Most of my changes are
> to the _read logic -- I also added a bunch of regression tests for
> the new features, but I did it in such a way that collision seems
> unlikely.  I'll check them in today or tomorrow and make sure
> everything is copacetic.

Eric, can you put off your checkin until after I've created the 2.3a1
release branch?

--Guido van Rossum (home page: http://www.python.org/~guido/)


From esr@snark.thyrsus.com  Tue Dec 31 15:27:58 2002
From: esr@snark.thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 10:27:58 -0500
Subject: [Python-Dev] ConfigParser changes
Message-ID: <200212311527.gBVFRwS9022537@snark.thyrsus.com>

I merged in my changes and regression-tested everything.  Looks
like we're OK, all tests passed including my new ones.

I also checked in a remnant doc change for some nntplib enhancements
I did a month or so ago.  This flushes my patch queue.  
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

"Government is not reason, it is not eloquence, it is force; like fire, a
troublesome servant and a fearful master. Never for a moment should it be left
to irresponsible action."
	-- George Washington, in a speech of January 7, 1790


From esr@thyrsus.com  Tue Dec 31 15:35:38 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 10:35:38 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <200212311522.gBVFMIY14143@odiug.zope.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com>
Message-ID: <20021231153538.GA22602@thyrsus.com>

Guido van Rossum <guido@python.org>:
> Eric, can you put off your checkin until after I've created the 2.3a1
> release branch?

Ouch.  Our mails crossed.  I just checked them in.  You can back them out
if you're feeling really paranoid -- but all the unit tests pass.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From guido@python.org  Tue Dec 31 15:50:45 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 10:50:45 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: Your message of "Tue, 31 Dec 2002 10:35:38 EST."
 <20021231153538.GA22602@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
Message-ID: <200212311550.gBVFojY09030@odiug.zope.com>

> Guido van Rossum <guido@python.org>:
> > Eric, can you put off your checkin until after I've created the 2.3a1
> > release branch?
> 
> Ouch.  Our mails crossed.  I just checked them in.  You can back them out
> if you're feeling really paranoid -- but all the unit tests pass.

I find it hard to believe that our mails crossed.  The first time I
asked you to wait until after the release was last night:

| Subject: Re: [Python-Dev] bugs in stdlib
| From: Guido van Rossum <guido@python.org>
| To: esr@thyrsus.com
| Date: Mon, 30 Dec 2002 20:58:25 -0500
| 
| > I've got another fix and some enhancements in the works for
| > ConfigParser.  Passes all regression tests, but I'm field-testing
| > before I commit.  Just because I'm nervous about patch collisions,
| > what code did your patch affect?
| 
| CVS is your friend, Eric. :-)
| 
| Can you put off checking in your changes until after the 2.3a1
| release tomorrow?  I'm a bit nervous about last-minute changes to
| ConfigParser, based on the history of that module.
| 
| --Guido van Rossum (home page: http://www.python.org/~guido/)

If you saw that and chose to ignore my advice, I would've appreciated
a reply at least.

I'll leave the decision to revert your checkin to Fred, who knows the
code (and has pending changes to it of his own, which now cause
conflicts).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From just@letterror.com  Tue Dec 31 15:58:48 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 31 Dec 2002 16:58:48 +0100
Subject: [Python-Dev] Python 2.3a1 release status
In-Reply-To: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <r01050400-1022-CEEB55001CD811D7A7C4003065D5E7E4@[10.0.0.3]>

Guido van Rossum wrote:

> If you know of other pending changes that haven't been committed,
> either check them in (only if they're stable, tested and documented,
> of course!) or let me know ASAP.

One thing I completely forgot to persue because of PEP 302 was the plan
to move Tools/freeze/modulefinder.py to Lib/. I would love for this to
be in 2.3a1, but nor Thomas Heller nor I have written the needed
documentation for it yet. If you allow it to go into 2.3a1, I promise to
submit documentation well before 2.3a2.

There's also a patch from me for modulefinder.py sitting on sf which
would be nice to make it in at the same time:
  http://www.python.org/sf/643711
It vastly reduces the number of false warnings about missing submodules.
Additionally it adds a new API for getting info about missing modules
which distinguishes between "surely missing" and "maybe missing".

Just


From guido@python.org  Tue Dec 31 16:03:38 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 11:03:38 -0500
Subject: [Python-Dev] Python 2.3a1 release status
In-Reply-To: Your message of "Tue, 31 Dec 2002 16:58:48 +0100."
 <r01050400-1022-CEEB55001CD811D7A7C4003065D5E7E4@[10.0.0.3]>
References: <r01050400-1022-CEEB55001CD811D7A7C4003065D5E7E4@[10.0.0.3]>
Message-ID: <200212311603.gBVG3cH16115@odiug.zope.com>

> One thing I completely forgot to persue because of PEP 302 was the plan
> to move Tools/freeze/modulefinder.py to Lib/. I would love for this to
> be in 2.3a1, but nor Thomas Heller nor I have written the needed
> documentation for it yet. If you allow it to go into 2.3a1, I promise to
> submit documentation well before 2.3a2.

Fine.  At least add a note to Misc/NEWS, and make sure that the freeze
tool still works.

> There's also a patch from me for modulefinder.py sitting on sf which
> would be nice to make it in at the same time:
>   http://www.python.org/sf/643711
> It vastly reduces the number of false warnings about missing submodules.
> Additionally it adds a new API for getting info about missing modules
> which distinguishes between "surely missing" and "maybe missing".

Ditto.  It looks like it's going to be at least a couple more hours
before datetime.c is ready anyway.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From esr@thyrsus.com  Tue Dec 31 16:03:47 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 11:03:47 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <200212311550.gBVFojY09030@odiug.zope.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com>
Message-ID: <20021231160347.GA22796@thyrsus.com>

Guido van Rossum <guido@python.org>:
> If you saw that and chose to ignore my advice, I would've appreciated
> a reply at least.

I did not.  I would have heeded that advice.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From lalo@laranja.org  Tue Dec 31 16:14:45 2002
From: lalo@laranja.org (Lalo Martins)
Date: Tue, 31 Dec 2002 14:14:45 -0200
Subject: [Python-Dev] sstruct
Message-ID: <20021231161445.GC31681@laranja.org>

While idly browsing around, I came across Just's module sstruct.  Any good
reason this is not in the library?  It is a kind of cross between struct and
ctypes, only a few years more mature.  With new-style classes it could
probably gain some nifty features.

http://starship.python.net/~just/code/sstruct.py

[]s,
                                               |alo
                                               +----
--
            Those who trade freedom for security
               lose both and deserve neither.
--
http://www.laranja.org/                mailto:lalo@laranja.org
         pgp key: http://www.laranja.org/pessoal/pgp

Eu jogo RPG! (I play RPG)         http://www.eujogorpg.com.br/
GNU: never give up freedom                 http://www.gnu.org/


From martin@v.loewis.de  Tue Dec 31 16:30:01 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 31 Dec 2002 17:30:01 +0100
Subject: [Python-Dev] sstruct
In-Reply-To: <20021231161445.GC31681@laranja.org>
References: <20021231161445.GC31681@laranja.org>
Message-ID: <3E11C609.5000206@v.loewis.de>

Lalo Martins wrote:
> While idly browsing around, I came across Just's module sstruct.  Any good
> reason this is not in the library?  

I can think of two reasons:
1. Nobody has proposed to include it, yet.
2. It is not obvious that it is useful.

On the latter point: What application is finding it so useful that they 
incorporate it in their distribution?

Regards,
Martin



From fdrake@acm.org  Tue Dec 31 16:31:47 2002
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 31 Dec 2002 11:31:47 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <200212311550.gBVFojY09030@odiug.zope.com>
References: <20021230234139.GE12063@epoch.metaslash.com>
 <20021231014355.GA14992@thyrsus.com>
 <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
 <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
 <200212311550.gBVFojY09030@odiug.zope.com>
Message-ID: <15889.50803.219337.935244@grendel.zope.com>

Guido van Rossum writes:
 > I'll leave the decision to revert your checkin to Fred, who knows the
 > code (and has pending changes to it of his own, which now cause
 > conflicts).

The conflict was minor; Eric managed to add a blank line at the end of
the file, and I'd made adjustments to the exception being raised
there.

Regarding Eric's changes, I'd like to see them discussed before being
made part of the library; the interface seems quite ad hoc, and it's
not clear that they are the right way to go about adding sequence
support.

What I've seen for sequences in INI files in the past is something
like this (working from memory):

  [SomeSection]
  NumPathEntries=3
  PathEntry1=c:\path\one
  PathEntry2=c:\path\two
  PathEntry3=c:\path\three

(I'm not advocating that as the "right" way to do it, I'm just noting
that what I've seen in practice is very different from the solution
Eric implemented.)

The value of having __str__() return a parsable INI file is highly
questionable; the motivation should be explained and discussed.  The
issue of ordering sections and options in the same way as the input
seems unimportant as well; comments are lost anyway.  Being able to
surgically edit a config file using the ConfigParser module is simply
not a supported use case.  (I wrote a module once that did support
surgical editing, and it was a royal pain!  Long lost in a disk
crash.)

I'll be reverting the change shortly.  Eric, I would like to encourage
you to pursue this functionality via discussion on python-dev with the
patch submitted via SourceForge.  The functional extensions are not
unwelcome by any means.

NB: For a very different approach to handling configuration, there's
something in Zope's CVS called "ZConfig".  Of particular interest is
the development branch that includes support for configuration schema:

http://cvs.zope.org/Packages/ZConfig/?only_with_tag=zconfig-schema-devel-branch

BTW, I'm back on python-dev now, but it's still a good idea to CC
things that need my specific attention.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation


From esr@thyrsus.com  Tue Dec 31 16:24:06 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 11:24:06 -0500
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <20021231160347.GA22796@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <20021231160347.GA22796@thyrsus.com>
Message-ID: <20021231162406.GA22850@thyrsus.com>

Eric S. Raymond <esr@thyrsus.com>:
> Guido van Rossum <guido@python.org>:
> > If you saw that and chose to ignore my advice, I would've appreciated
> > a reply at least.
> 
> I did not.  I would have heeded that advice.

On reflection, more explanation seems called for...

I've been flooded in a particularly repellent way recently.  Somebody
is using my From address for spam mail about quack remedies for
prostate trouble, somebody so illiterate that the Subject line read
"urnination flow".  As a result, I've been getting several *thousand*
bounce messages a day from sites rightly rejecting this crud, and
hitting the 'd' key in rapid-fire mode.  

I may have inadvertently flushed your mail, dammit, which is not
something I would ever do deliberately.  I subjunctively apologize.
You may not be alone, I fear.  I almost blew past mail from Dennis
Ritchie last night, just barely caught myself in time.  I'm wondering
now what other important bits I lost... :-( :-( :-(

I do not regard the ConfigParser changes as at all urgent -- the bug is
a minor one.  So I won't be offended if Fred chooses to back them out.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From skip@pobox.com  Tue Dec 31 16:44:55 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 31 Dec 2002 10:44:55 -0600
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <20021231162406.GA22850@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com>
 <20021231014355.GA14992@thyrsus.com>
 <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
 <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
 <200212311550.gBVFojY09030@odiug.zope.com>
 <20021231160347.GA22796@thyrsus.com>
 <20021231162406.GA22850@thyrsus.com>
Message-ID: <15889.51591.384682.949437@montanaro.dyndns.org>

    Eric> I've been flooded in a particularly repellent way recently.
    Eric> Somebody is using my From address for spam mail about quack
    Eric> remedies for prostate trouble, somebody so illiterate that the
    Eric> Subject line read "urnination flow".  As a result, I've been
    Eric> getting several *thousand* bounce messages a day from sites
    Eric> rightly rejecting this crud, and hitting the 'd' key in rapid-fire
    Eric> mode.

Sounds like you need to install spambayes or bogofilter. ;-)

Skip


From barry@python.org  Tue Dec 31 16:51:54 2002
From: barry@python.org (Barry A. Warsaw)
Date: Tue, 31 Dec 2002 11:51:54 -0500
Subject: [Python-Dev] bugs in stdlib
References: <20021230234139.GE12063@epoch.metaslash.com>
 <20021231014355.GA14992@thyrsus.com>
 <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
 <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
 <200212311550.gBVFojY09030@odiug.zope.com>
 <20021231160347.GA22796@thyrsus.com>
 <20021231162406.GA22850@thyrsus.com>
Message-ID: <15889.52010.949501.208196@gargle.gargle.HOWL>

>>>>> "ESR" == Eric S Raymond <esr@thyrsus.com> writes:

    ESR> I've been flooded in a particularly repellent way recently.
    ESR> Somebody is using my From address for spam mail about quack
    ESR> remedies for prostate trouble, somebody so illiterate that
    ESR> the Subject line read "urnination flow".

Welcome to the world of Klez.  Someone who has you in their address
book is infected.

-Barry


From skip@pobox.com  Tue Dec 31 17:04:24 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 31 Dec 2002 11:04:24 -0600
Subject: [Python-Dev] Patch 640843 - split OPT into two parts
Message-ID: <15889.52760.92971.951623@montanaro.dyndns.org>

I took a break from my fiddling with the OPT make/configure variable when I
discovered the problems with out-of-tree builds.  Andrew got that fixed a
couple days ago, so I revisited my changes.  I'd like to check them in, but
should probably wait until Guido bundles 2.3a1.  Changes are to these files:

    configure.in (and configure, of course)
    setup.py
    Makefile.pre.in
    Misc/NEWS

There are some structural changes in configure.in that make it worth a
careful look-see.  In particular, I hope I haven't bolluxed up Martin's SCO
changes from September (I can't test on that platform).

Here's the new text from Misc/NEWS:

- On systems which build using the configure script, compiler flags which
  used to be lumped together using the OPT flag have been split into two
  groups, OPT and BASECFLAGS.  OPT is meant to carry just optimization- and
  debug-related flags like "-g" and "-O3".  BASECFLAGS is meant to carry
  compiler flags that are required to get a clean compile.  On some
  platforms (many Linux flavors in particular) BASECFLAGS will be empty by
  default.  On others, such as MacOSX and SCO, it will contain required
  flags.  This change allows people building Python to override OPT without
  fear of clobbering compiler flags which are required to get a clean build.

Everything passes on both MacOSX and Mandrake 8.1 from CVS as of a few
minutes ago.

I uploaded a new context diff to SF so people can eyeball it, though I'll
wait for Guido to say "check it in".  There's one itty-bitty change to
setup.py that's unrelated.  It adds the fink directories /sw/lib and
/sw/include to the relevant search lists when building on Darwin.

Skip


From esr@thyrsus.com  Tue Dec 31 16:57:01 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 11:57:01 -0500
Subject: [Python-Dev] ConfigParser changes
In-Reply-To: <15889.50803.219337.935244@grendel.zope.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com>
Message-ID: <20021231165701.GA22958@thyrsus.com>

Fred L. Drake, Jr. <fdrake@acm.org>:
> Regarding Eric's changes, I'd like to see them discussed before being
> made part of the library; the interface seems quite ad hoc, and it's
> not clear that they are the right way to go about adding sequence
> support.

Do you have the same reaction to the multiline string support?

And do you want the line-continuation bugfix?

> The value of having __str__() return a parsable INI file is highly
> questionable; the motivation should be explained and discussed.

Cleanliness.  I believe in invertible representations.  Whenever I write
a class, I like to have its str() or repr() emit something parseable.
If nothing else, this is useful for debugging.  This is just general
principles, I don't (unlike my other changes) have a use case for it.

> The issue of ordering sections and options in the same way as the input
> seems unimportant as well; comments are lost anyway.  Being able to
> surgically edit a config file using the ConfigParser module is simply
> not a supported use case.

It is now.  That was the point of the enhancements.

> I'll be reverting the change shortly.  Eric, I would like to encourage
> you to pursue this functionality via discussion on python-dev with the
> patch submitted via SourceForge.  The functional extensions are not
> unwelcome by any means.

OK.  Then let the discussion begin.  Here are the issues:

1. There is a minor bug in the way continuations are handled that, in
some circumstances, can mean the write() representation is not invertible.

2. Currently ConfigParser cannot support multiline strings -- that is, values
that are multiline and have embedded whitespace.  I need this capability.
I added support for this through a class member that lets you declare string 
quotes.  Is there some objection to supporting this value type, or just to
the magic-class-member interface?

3. Currently ConfigParser cannot support structured values. Again, I
need this capability (for association lists of coordinates and names,
as it happens).  The syntax I have implemented is a configurable list
bracket character surrounding comma-separated lists.  The alternative
Fred suggests is, I think, extremely ugly.  If anyone has a more
natural suggestion than my proposal, I'm willing to hear it.

4. I *do* in fact want to support `surgical' alteration of .INI files.
I have a use case for this in a configuration editor I'm writing for
FreeCiv.  More generally, when writing code to support invertible
representations, I think it is good citizenship to perturb the
original data as little as possible.  Thus I regard the fact that 
the present code permutes options and sections into an arbitrary 
order dependent on the hash table implementation as a design bug, and
actually a fairly serious sort of thoughtlessness.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From guido@python.org  Tue Dec 31 17:13:56 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 12:13:56 -0500
Subject: [Python-Dev] ConfigParser changes
In-Reply-To: Your message of "Tue, 31 Dec 2002 11:57:01 EST."
 <20021231165701.GA22958@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com>
 <20021231165701.GA22958@thyrsus.com>
Message-ID: <200212311713.gBVHDvJ24099@odiug.zope.com>

> Cleanliness.  I believe in invertible representations.  Whenever I
> write a class, I like to have its str() or repr() emit something
> parseable.  If nothing else, this is useful for debugging.  This is
> just general principles, I don't (unlike my other changes) have a
> use case for it.

I don't like classes whose str() or repr() can be many lines in a
typical case.  Since this is read from a file, there should be (and
is) a method to write it to a file; if there's no use case for
converting the whole shebang to a string, don't provide that, and
especially not as str() or repr().

--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido@python.org  Tue Dec 31 17:26:26 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 12:26:26 -0500
Subject: [Python-Dev] Patch 640843 - split OPT into two parts
In-Reply-To: Your message of "Tue, 31 Dec 2002 11:04:24 CST."
 <15889.52760.92971.951623@montanaro.dyndns.org>
References: <15889.52760.92971.951623@montanaro.dyndns.org>
Message-ID: <200212311726.gBVHQQs24193@odiug.zope.com>

> I took a break from my fiddling with the OPT make/configure variable when I
> discovered the problems with out-of-tree builds.  Andrew got that fixed a
> couple days ago, so I revisited my changes.  I'd like to check them in, but
> should probably wait until Guido bundles 2.3a1.  Changes are to these files:
> 
>     configure.in (and configure, of course)
>     setup.py
>     Makefile.pre.in
>     Misc/NEWS
> 
> There are some structural changes in configure.in that make it worth a
> careful look-see.  In particular, I hope I haven't bolluxed up Martin's SCO
> changes from September (I can't test on that platform).
> 
> Here's the new text from Misc/NEWS:
> 
> - On systems which build using the configure script, compiler flags which
>   used to be lumped together using the OPT flag have been split into two
>   groups, OPT and BASECFLAGS.  OPT is meant to carry just optimization- and
>   debug-related flags like "-g" and "-O3".  BASECFLAGS is meant to carry
>   compiler flags that are required to get a clean compile.  On some
>   platforms (many Linux flavors in particular) BASECFLAGS will be empty by
>   default.  On others, such as MacOSX and SCO, it will contain required
>   flags.  This change allows people building Python to override OPT without
>   fear of clobbering compiler flags which are required to get a clean build.
> 
> Everything passes on both MacOSX and Mandrake 8.1 from CVS as of a few
> minutes ago.
> 
> I uploaded a new context diff to SF so people can eyeball it, though I'll
> wait for Guido to say "check it in".  There's one itty-bitty change to
> setup.py that's unrelated.  It adds the fink directories /sw/lib and
> /sw/include to the relevant search lists when building on Darwin.

Are you sure you want to risk this right before the release?  If it
breaks some platform's build, we're hosed.

OTOH if you want to risk it, I'm not going to stop you -- I agree that
in principle the change is a desirable one.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From python@rcn.com  Tue Dec 31 17:32:18 2002
From: python@rcn.com (Raymond Hettinger)
Date: Tue, 31 Dec 2002 12:32:18 -0500
Subject: [Python-Dev] test_coercion
Message-ID: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer>

Can anyone reproduce this test failure:

C:\py23\Lib\test>python_d regrtest.py test_coercion
test_coercion
test test_coercion produced unexpected output:
**********************************************************************
*** mismatch between line 523 of expected output and line 523 of actual output:
- [1] *= <MethodNumber 1> => [1]
+ [1] *= <MethodNumber 1> ... exceptions.TypeError
*** mismatch between line 535 of expected output and line 535 of actual output:
- [1] *= <CoerceNumber 2> => [1, 1]
+ [1] *= <CoerceNumber 2> ... exceptions.TypeError
***********************************************************************
1 test failed:
    test_coercion
[12762 refs]



From esr@thyrsus.com  Tue Dec 31 17:26:42 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 12:26:42 -0500
Subject: [Python-Dev] ConfigParser changes
In-Reply-To: <200212311713.gBVHDvJ24099@odiug.zope.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <200212311713.gBVHDvJ24099@odiug.zope.com>
Message-ID: <20021231172642.GC22958@thyrsus.com>

Guido van Rossum <guido@python.org>:
> I don't like classes whose str() or repr() can be many lines in a
> typical case.  Since this is read from a file, there should be (and
> is) a method to write it to a file; if there's no use case for
> converting the whole shebang to a string, don't provide that, and
> especially not as str() or repr().

Noted for the futuee, thanks.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From fdrake@acm.org  Tue Dec 31 17:36:15 2002
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 31 Dec 2002 12:36:15 -0500
Subject: [Python-Dev] Re: ConfigParser changes
In-Reply-To: <20021231165701.GA22958@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com>
 <20021231014355.GA14992@thyrsus.com>
 <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
 <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
 <200212311550.gBVFojY09030@odiug.zope.com>
 <15889.50803.219337.935244@grendel.zope.com>
 <20021231165701.GA22958@thyrsus.com>
Message-ID: <15889.54671.243224.307879@grendel.zope.com>

Eric S. Raymond writes:
 > Do you have the same reaction to the multiline string support?

Yes.  It should definately be discussed.

 > And do you want the line-continuation bugfix?

That would be nice to have; I should have responded to that point
separately.  Feel free to commit a minimal patch to fix that, and add
a regression test to Lib/test/test_cfgparser.py.

 > > The value of having __str__() return a parsable INI file is highly
 > > questionable; the motivation should be explained and discussed.
 > 
 > Cleanliness.  I believe in invertible representations.  Whenever I write
 > a class, I like to have its str() or repr() emit something parseable.
 > If nothing else, this is useful for debugging.  This is just general
 > principles, I don't (unlike my other changes) have a use case for it.

I don't think it's clear that using __str__() for this is valuable.
Creating a potentially large string in memory may not be the best
approach; I actually like the write() method better; the caller can
decide to use a StringIO if that's what makes sense, or toss things to
a real file if needed.

If others like this, I won't object.  It should be a separate patch
from the other changes, though.

 > > The issue of ordering sections and options in the same way as the input
 > > seems unimportant as well; comments are lost anyway.  Being able to
 > > surgically edit a config file using the ConfigParser module is simply
 > > not a supported use case.
 > 
 > It is now.  That was the point of the enhancements.

Perhaps I missed it; did you preserve comments as well?

 > OK.  Then let the discussion begin.  Here are the issues:

Excellent!  Thanks.

 > 1. There is a minor bug in the way continuations are handled that, in
 > some circumstances, can mean the write() representation is not invertible.

Per above, that can be fixed at any time.  Bugs are written to be
squashed.  ;-)

 > 2. Currently ConfigParser cannot support multiline strings -- that
 > is, values that are multiline and have embedded whitespace.  I need
 > this capability.  I added support for this through a class member
 > that lets you declare string quotes.  Is there some objection to
 > supporting this value type, or just to the magic-class-member
 > interface?

I think the interface should be discussed; your implementation may be
sufficient, but after working with pyexpat as much as I have, I've
developed a strong dislike for attributes that have substantial side
effects.

If we're going to add a special syntax for multi-line values, we may
need to think about encoding special values, text encoding, and
Unicode.  And that's never easy to get concensus on.  ;-)

 > 3. Currently ConfigParser cannot support structured values. Again, I
 > need this capability (for association lists of coordinates and names,
 > as it happens).  The syntax I have implemented is a configurable list
 > bracket character surrounding comma-separated lists.  The alternative
 > Fred suggests is, I think, extremely ugly.  If anyone has a more
 > natural suggestion than my proposal, I'm willing to hear it.

I think "suggests" is too strong a word; I was only citing precedence,
not advocating that approach.  I think it's pretty ugly as well.

 > 4. I *do* in fact want to support `surgical' alteration of .INI files.
 > I have a use case for this in a configuration editor I'm writing for
 > FreeCiv.  More generally, when writing code to support invertible
 > representations, I think it is good citizenship to perturb the
 > original data as little as possible.  Thus I regard the fact that

I think there's a distinction between reading a configuration and
editing it:  The original code never wrote a file back out.  That was
something you added in revision 1.19.

 > the present code permutes options and sections into an arbitrary 
 > order dependent on the hash table implementation as a design bug, and
 > actually a fairly serious sort of thoughtlessness.

Only if you write the file back out.  ;-)

Seriously, I have no objection to supporting surgical editing.  I do
think that separate classes should be used for read-only uses and
read-write uses; a read-only ConfigParser-thingie should remain very
lightweight, and surgical editing just isn't that.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation


From bellman+pep-divmod@lysator.liu.se  Tue Dec 31 17:38:37 2002
From: bellman+pep-divmod@lysator.liu.se (Thomas Bellman)
Date: Tue, 31 Dec 2002 18:38:37 +0100 (MET)
Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors
Message-ID: <200212311738.SAA02244@proton.lysator.liu.se>

I humbly submit this PEP for your dissection.

As I am not subscribed to python-dev, please make sure that your
comments are CC:ed to <bellman+pep-divmod@lysator.liu.se>, so I
can see them.

I have also posted this to comp.lang.python.

I have written an implementation also, but I need to check it
some more to see if I've got the reference counting correct
before I dare post it. :-)


PEP: 303
Title: Extend divmod() for Multiple Divisors
Version: $Revision: 1.2 $
Last-Modified: $Date: 2002/12/31 16:02:49 $
Author: Thomas Bellman <bellman+pep-divmod@lysator.liu.se>
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 31-Dec-2002
Python-Version: 2.3
Post-History:


Abstract

    This PEP describes an extension to the built-in divmod() function,
    allowing it to take multiple divisors, chaining several calls to
    divmod() into one.


Specification

    The built-in divmod() function would be changed to accept multiple
    divisors, changing its signature from divmod(dividend, divisor) to
    divmod(dividend, *divisors).  The dividend is divided by the last
    divisor, giving a quotient and a remainder.  The quotient is then
    divided by the second to last divisor, giving a new quotient and
    remainder.  This is repeated until all divisors have been used,
    and divmod() then returns a tuple consisting of the quotient from
    the last step, and the remainders from all the steps.

    A Python implementation of the new divmod() behaviour could look
    like:

        def divmod(dividend, *divisors):
            modulos = ()
            q = dividend
            while divisors:
                q,r = q.__divmod__(divisors[-1])
                modulos = (r,) + modulos
                divisors = divisors[:-1]
            return (q,) + modulos


Motivation

    Occasionally one wants to perform a chain of divmod() operations,
    calling divmod() on the quotient from the previous step, with
    varying divisors.  The most common case is probably converting a
    number of seconds into weeks, days, hours, minutes and seconds.
    This would today be written as:

        def secs_to_wdhms(seconds):
            m,s = divmod(seconds, 60)
            h,m = divmod(m, 60)
            d,h = divmod(h, 24)
            w,d = divmod(d, 7)
            return (w,d,h,m,s)

    This is tedious and easy to get wrong each time you need it.

    If instead the divmod() built-in is changed according the proposal,
    the code for converting seconds to weeks, days, hours, minutes and
    seconds then become

        def secs_to_wdhms(seconds):
            w,d,h,m,s = divmod(seconds, 7, 24, 60, 60)
            return (w,d,h,m,s)

    which is easier to type, easier to type correctly, and easier to
    read.

    Other applications are:

    - Astronomical angles (declination is measured in degrees, minutes
      and seconds, right ascension is measured in hours, minutes and
      seconds).
    - Old British currency (1 pound = 20 shilling, 1 shilling = 12 pence)
    - Anglo-Saxon length units: 1 mile = 1760 yards, 1 yard = 3 feet,
      1 foot = 12 inches.
    - Anglo-Saxon weight units: 1 long ton = 160 stone, 1 stone = 14
      pounds, 1 pound = 16 ounce, 1 ounce = 16 dram
    - British volumes: 1 gallon = 4 quart, 1 quart = 2 pint, 1 pint
      = 20 fluid ounces


Rationale

    The idea comes from APL, which has an operator that does this.  (I
    don't remember what the operator looks like, and it would probably
    be impossible to render in ASCII anyway.)

    The APL operator takes a list as its second operand, while this
    PEP proposes that each divisor should be a separate argument to
    the divmod() function.  This is mainly because it is expected that
    the most common uses will have the divisors as constants right in
    the call (as the 7, 24, 60, 60 above), and adding a set of
    parentheses or brackets would just clutter the call.

    Requiring an explicit sequence as the second argument to divmod()
    would seriously break backwards compatibility.  Making divmod()
    check its second argument for being a sequence is deemed to be too
    ugly to contemplate.  And in the case where one *does* have a
    sequence that is computed other-where, it is easy enough to write
    divmod(x, *divs) instead.

    Requiring at least one divisor, i.e rejecting divmod(x), has been
    considered, but no good reason to do so has come to mind, and is
    thus allowed in the name of generality.

    Calling divmod() with no divisors should still return a tuple (of
    one element).  Code that calls divmod() with a varying number of
    divisors, and thus gets a return value with an "unknown" number of
    elements, would otherwise have to special case that case.  Code
    that *knows* it is calling divmod() with no divisors is considered
    to be too silly to warrant a special case.

    Processing the divisors in the other direction, i.e dividing with
    the first divisor first, instead of dividing with the last divisor
    first, has been considered.  However, the result comes with the
    most significant part first and the least significant part last
    (think of the chained divmod as a way of splitting a number into
    "digits", with varying weights), and it is reasonable to specify
    the divisors (weights) in the same order as the result.

    The inverse operation:

        def inverse_divmod(seq, *factors):
            product = seq[0]
            for x,y in zip(factors, seq[1:]):
                product = product * x + y
            return product

    could also be useful.  However, writing

        seconds = (((((w * 7) + d) * 24 + h) * 60 + m) * 60 + s)

    is less cumbersome both to write and to read than the chained
    divmods.  It is therefore deemed to be less important, and its
    introduction can be deferred to its own PEP.  Also, such a
    function needs a good name, and the PEP author has not managed to
    come up with one yet.

    Calling divmod("spam") does not raise an error, despite strings
    supporting neither division nor modulo.  However, unless we know
    the other object too, we can't determine whether divmod() would
    work or not, and thus it seems silly to forbid it.


Backwards Compatibility

    Any module that replaces the divmod() function in the __builtin__
    module, may cause other modules using the new syntax to break.  It
    is expected that this is very uncommon.

    Code that expects a TypeError exception when calling divmod() with
    anything but two arguments will break.  This is also expected to
    be very uncommon.

    No other issues regarding backwards compatibility are known.


Reference Implementation

    Not finished yet, but it seems a rather straightforward
    new implementation of the function builtin_divmod() in
    Python/bltinmodule.c


Copyright

    This document has been placed in the public domain.


-- 
Thomas Bellman,   Lysator Computer Club,   Linköping University,  Sweden
"Adde parvum parvo magnus acervus erit"       ! bellman @ lysator.liu.se
          (From The Mythical Man-Month)       ! Make Love -- Nicht Wahr!


From Jack.Jansen@oratrix.com  Tue Dec 31 17:38:49 2002
From: Jack.Jansen@oratrix.com (Jack Jansen)
Date: Tue, 31 Dec 2002 18:38:49 +0100
Subject: [Python-Dev] Python 2.3a1 release status
In-Reply-To: <200212311402.gBVE2XR14339@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <B847823E-1CE6-11D7-B7DE-000A27B19B96@oratrix.com>

On dinsdag, dec 31, 2002, at 15:02 Europe/Amsterdam, Guido van Rossum 
wrote:
> Jack Jansen: I don't expect that you're quite ready with the Mac tree.
> Or are you?  Let me know what you want me to say about the Mac in the
> announcement and on the web.

Sorry for not telling you earlier, but all the Mac stuff (the 
unix-Mac-stuff, that is) has been ready since last night. MacPython-OS9 
will take another week (or more), so I'd like to get control of the 
release branch again after the unix/win release has been done.

The main bit of Mac news is that all the old MacPython tools and 
modules now work (if you do a framework build). I plan to do a binary 
release later (not for a1, probably starting with the beta's) and that 
will be called MacPython-OSX or MacPython for short (MacPython-OS9 will 
be the new name for the old MacPython).
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -



From just@letterror.com  Tue Dec 31 17:46:28 2002
From: just@letterror.com (Just van Rossum)
Date: Tue, 31 Dec 2002 18:46:28 +0100
Subject: [Python-Dev] sstruct
In-Reply-To: <3E11C609.5000206@v.loewis.de>
Message-ID: <r01050400-1022-F262B8D41CE711D7A7C4003065D5E7E4@[10.0.0.3]>

"Martin v. L=F6wis" wrote:

> Lalo Martins wrote:
> > While idly browsing around, I came across Just's module sstruct.=20
> > Any good reason this is not in the library? =20
>=20
> I can think of two reasons:
> 1. Nobody has proposed to include it, yet.

Including its author ;-)

> 2. It is not obvious that it is useful.
>=20
> On the latter point: What application is finding it so useful that
> they incorporate it in their distribution?

It's incredibly useful in my TrueType/OpenType conversion package
(http://fonttools.sf.net/) where I convert lots of arbitrary and largely
unrelated struct-like data formats to and from Python objects. Having
the field names in the format string and automatically using these for
dict keys or object attribute names was a major win in maintainability
and readability of the code, and it being only a thin layer on top of
the struct module makes it pretty fast.

That said, I was never convinced of a usability so wide as to propose it
for the std lib, and the amount of feedback I got over the years (it's
pretty old!) didn't change that in a positive direction.

The version on sf might be slightly newer than the one on starship (I
don't even know!), but even on sf it hasn't changed since I checked it
in there, three years ago:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/fonttools/fonttools/Lib/
sstruct.py

Just


From skip@pobox.com  Tue Dec 31 18:11:16 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 31 Dec 2002 12:11:16 -0600
Subject: [Python-Dev] Patch 640843 - split OPT into two parts
In-Reply-To: <200212311726.gBVHQQs24193@odiug.zope.com>
References: <15889.52760.92971.951623@montanaro.dyndns.org>
 <200212311726.gBVHQQs24193@odiug.zope.com>
Message-ID: <15889.56772.30091.47277@montanaro.dyndns.org>

    >> I'd like to check them in, but should probably wait until Guido
    >> bundles 2.3a1.
    ...

    Guido> Are you sure you want to risk this right before the release?  If
    Guido> it breaks some platform's build, we're hosed.

That's why I said I should probably wait until 2.3a1 is out the door. ;-)

Skip


From guido@python.org  Tue Dec 31 18:22:19 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 13:22:19 -0500
Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors
In-Reply-To: Your message of "Tue, 31 Dec 2002 18:38:37 +0100."
 <200212311738.SAA02244@proton.lysator.liu.se>
References: <200212311738.SAA02244@proton.lysator.liu.se>
Message-ID: <200212311822.gBVIMJM24712@odiug.zope.com>

On the one hand I find this a nice backwards-compatible extension of
divmod().  On the other hand I've coded this use case with separate
divmod() calls and never had any trouble getting it right.  So I think
this doesn't address a real need, and it conflicts with the need to
avoid unnecessary growth of the language.  Explaining it (especially
why it works backward) takes some effort, and I don't want to do that
to all the book authors etc.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From lalo@laranja.org  Tue Dec 31 18:44:57 2002
From: lalo@laranja.org (Lalo Martins)
Date: Tue, 31 Dec 2002 16:44:57 -0200
Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors
In-Reply-To: <200212311738.SAA02244@proton.lysator.liu.se>
References: <200212311738.SAA02244@proton.lysator.liu.se>
Message-ID: <20021231184457.GE31681@laranja.org>

I'm +0 on this (reasons below), but since Guido already said no:

I find this function pretty useful, as it serves to collapse ugly code, and
I hate ugly python code.  However, the name sucks.  With your changes,
'divmod' doesn't describe it anymore.

So I suggest you come up with a better name and offer it as a contribution
to, say, the math module.  In that case I would prefer to have the reverse
function too.

(The fact that using it with 2 args is equal to calling divmod is at most a
funny fact.  As Guido says, growing the library = good; growing the language =
questionable)

[]s,
                                               |alo
                                               +----
--
            Those who trade freedom for security
               lose both and deserve neither.
--
http://www.laranja.org/                mailto:lalo@laranja.org
         pgp key: http://www.laranja.org/pessoal/pgp

Eu jogo RPG! (I play RPG)         http://www.eujogorpg.com.br/
GNU: never give up freedom                 http://www.gnu.org/


From guido@python.org  Tue Dec 31 18:39:17 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 13:39:17 -0500
Subject: [Python-Dev] test_coercion
In-Reply-To: Your message of "Tue, 31 Dec 2002 12:32:18 EST."
 <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer>
References: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer>
Message-ID: <200212311839.gBVIdHm31228@odiug.zope.com>

> Can anyone reproduce this test failure:
> 
> C:\py23\Lib\test>python_d regrtest.py test_coercion
> test_coercion
> test test_coercion produced unexpected output:
> **********************************************************************
> *** mismatch between line 523 of expected output and line 523 of actual output:
> - [1] *= <MethodNumber 1> => [1]
> + [1] *= <MethodNumber 1> ... exceptions.TypeError
> *** mismatch between line 535 of expected output and line 535 of actual output:
> - [1] *= <CoerceNumber 2> => [1, 1]
> + [1] *= <CoerceNumber 2> ... exceptions.TypeError
> ***********************************************************************
> 1 test failed:
>     test_coercion
> [12762 refs]

It doesn't fail for me with current CVS on Linux and Windows 98.  I
tried the debug and the regular build.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From lalo@laranja.org  Tue Dec 31 18:31:17 2002
From: lalo@laranja.org (Lalo Martins)
Date: Tue, 31 Dec 2002 16:31:17 -0200
Subject: [Python-Dev] sstruct
In-Reply-To: <r01050400-1022-F262B8D41CE711D7A7C4003065D5E7E4@[10.0.0.3]>
References: <3E11C609.5000206@v.loewis.de> <r01050400-1022-F262B8D41CE711D7A7C4003065D5E7E4@[10.0.0.3]>
Message-ID: <20021231183117.GD31681@laranja.org>

On Tue, Dec 31, 2002 at 06:46:28PM +0100, Just van Rossum wrote:
> "Martin v. Löwis" wrote:
> > 1. Nobody has proposed to include it, yet.
> 
> Including its author ;-)

That is why I posted to a forum where I knew said author would read me ;-)

> > 2. It is not obvious that it is useful.
> > 
> > On the latter point: What application is finding it so useful that
> > they incorporate it in their distribution?

Heck, I would have used it instead of struct in all places I ever used
struct, if I knew about it before :-)

(But the only one of these who is public is the Python client library of the
PicoGUI toolkit.  It has a heavy dependency on struct, which could be made
much more readable with sstruct - actually this is why sstruct got my
attention in the first place.)

I guess it's essentialy Just's fault for not publicizing his cool stuff
better...

Perhaps it is not a good time for it to go into 2.3, but I think this is the
kind of thing that belongs in the library.  It does the same job as struct,
but with a more readable (and arguably, pythonic) API.  I can volunteer to
write the docs and tests, if needed.

[]s,
                                               |alo
                                               +----
--
            Those who trade freedom for security
               lose both and deserve neither.
--
http://www.laranja.org/                mailto:lalo@laranja.org
         pgp key: http://www.laranja.org/pessoal/pgp

Eu jogo RPG! (I play RPG)         http://www.eujogorpg.com.br/
GNU: never give up freedom                 http://www.gnu.org/


From esr@thyrsus.com  Tue Dec 31 18:31:02 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 13:31:02 -0500
Subject: [Python-Dev] Re: ConfigParser changes
In-Reply-To: <15889.54671.243224.307879@grendel.zope.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com>
Message-ID: <20021231183102.GA23564@thyrsus.com>

Fred L. Drake, Jr. <fdrake@acm.org>:
> That would be nice to have; I should have responded to that point
> separately.  Feel free to commit a minimal patch to fix that, and add
> a regression test to Lib/test/test_cfgparser.py.

On my to-do list.
 
> I don't think it's clear that using __str__() for this is valuable.
> Creating a potentially large string in memory may not be the best
> approach; I actually like the write() method better; the caller can
> decide to use a StringIO if that's what makes sense, or toss things to
> a real file if needed.

Guido has already made this point.

As a matter of taste, I like the serialization operation to be
separate from I/O.  I think that's cleaner, and in this case the
in-memory string is unlikely to get large.  But I don't actually need
this change, so I won't argue for it against opposition.

> Perhaps I missed it; did you preserve comments as well?

No.  But comments are fair game to be discarded, since they're not part
of the data content.  In my use case they're not important.
  
> I think the interface should be discussed; your implementation may be
> sufficient, but after working with pyexpat as much as I have, I've
> developed a strong dislike for attributes that have substantial side
> effects.

A fair point.  I would be happy to change the interface away from a magic
member to a real method.  Likewise for the structured-value stuff.
 
> If we're going to add a special syntax for multi-line values, we may
> need to think about encoding special values, text encoding, and
> Unicode.  And that's never easy to get concensus on.  ;-)

Then we shouldn't try.  Let's not let the ideal be the enemy of the good
here; the multiline-literal syntax is othorgonal to how we handle 
string encoding, and internationalization can be layered on it later.
 
>  > 3. Currently ConfigParser cannot support structured values. Again, I
>  > need this capability (for association lists of coordinates and names,
>  > as it happens).  The syntax I have implemented is a configurable list
>  > bracket character surrounding comma-separated lists.  The alternative
>  > Fred suggests is, I think, extremely ugly.  If anyone has a more
>  > natural suggestion than my proposal, I'm willing to hear it.
> 
> I think "suggests" is too strong a word; I was only citing precedence,
> not advocating that approach.  I think it's pretty ugly as well.

I'm taking suggestions on how to do it right.  I don't think there's any
obviously better way than what I've implemented, except maybe for making the
trigger a real method rather than a magic member.

>  > 4. I *do* in fact want to support `surgical' alteration of .INI files.
>  > I have a use case for this in a configuration editor I'm writing for
>  > FreeCiv.  More generally, when writing code to support invertible
>  > representations, I think it is good citizenship to perturb the
>  > original data as little as possible.  Thus I regard the fact that
> 
> I think there's a distinction between reading a configuration and
> editing it:  The original code never wrote a file back out.  That was
> something you added in revision 1.19.

Right, because I needed it.
 
> Seriously, I have no objection to supporting surgical editing.  I do
> think that separate classes should be used for read-only uses and
> read-write uses; a read-only ConfigParser-thingie should remain very
> lightweight, and surgical editing just isn't that.

I had this conversation with Guido a couple years back.  I too would have
been happier with two classes, one read-only and one that supports writing
back out.  Here is whatt Guido said and I replied:

---------------------------------------------------------------------------
Guido van Rossum <guido@beopen.com>:
> One possible suggestion (just for your contemplation): if you're only
> adding methods, would it make sense to make a subclass, so that from
> the class one uses it is clear whether one intends to modify the
> options or just to read them?  That would enable future optimizations.

Yes, and I even know what I'd call the class: ConfigEditor.

Unfortunately this plan it trips on a historical fact; one (1) of the
write methods, add_section(), already existed in ConfigParser.  Moving
it to a ConfigEditor subclass would break backward compatibility.  And
I think it would be just too ugly to move all but one of the write
methods into ConfigEditor and leave add_section() in the base class.

If not for this problem I would be more than happy to do as you suggest.
---------------------------------------------------------------------------

Guido dropped the thread, which I took to mean that he saw the force
of the objection.  But I'd *still* like to do as he suggested.  

If the consensus is that its OK to deprecate add_section() and write()
in the base class, I'll write ConfigEditor and move all my surgery
stuff over there in a heartbeat.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From DavidA@ActiveState.com  Tue Dec 31 18:38:38 2002
From: DavidA@ActiveState.com (David Ascher)
Date: Tue, 31 Dec 2002 10:38:38 -0800
Subject: [Python-Dev] bugs in stdlib
In-Reply-To: <20021231162406.GA22850@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com> <20021231014355.GA14992@thyrsus.com> <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <20021231160347.GA22796@thyrsus.com> <20021231162406.GA22850@thyrsus.com>
Message-ID: <3E11E42E.8080506@ActiveState.com>

Eric S. Raymond wrote:
> Eric S. Raymond <esr@thyrsus.com>:
> 
>>Guido van Rossum <guido@python.org>:
>>
>>>If you saw that and chose to ignore my advice, I would've appreciated
>>>a reply at least.
>>
>>I did not.  I would have heeded that advice.
> 
> 
> On reflection, more explanation seems called for...
> 
> I've been flooded in a particularly repellent way recently.  Somebody
> is using my From address for spam mail about quack remedies for
> prostate trouble, somebody so illiterate that the Subject line read
> "urnination flow".  

We've been seeing misspellings as deliberate ways to get around word-list 
filters.  It's probably not illiteracy as much as wickedness.






From guido@python.org  Tue Dec 31 18:44:18 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 13:44:18 -0500
Subject: [Python-Dev] Creating 2.3a1 branch
Message-ID: <200212311844.gBVIiID05084@odiug.zope.com>

I'm creating the 2.3a1 release branch.  Checkins from now on won't
make it into 2.3a1 unless you mail me ASAP for special dispensation.
(Tim is tracking down a leak that may get this.)

The branch point is tagged with r23a1-fork; the branch itself will be
called r23a1-branch.  Thanks everybody!  Release later this afternoon.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From nas@python.ca  Tue Dec 31 19:04:23 2002
From: nas@python.ca (Neil Schemenauer)
Date: Tue, 31 Dec 2002 11:04:23 -0800
Subject: [Python-Dev] test_coercion
In-Reply-To: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer>
References: <003901c2b0f2$ad0d4f20$125ffea9@oemcomputer>
Message-ID: <20021231190423.GA749@glacier.arctrix.com>

Raymond Hettinger wrote:
> Can anyone reproduce this test failure:

Not I.  That behavior change was introduced recently by me.  The main
change was abstract.c 2.113.  Are you sure python_d is up-to-date?

  Neil


From fdrake@acm.org  Tue Dec 31 19:01:54 2002
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 31 Dec 2002 14:01:54 -0500
Subject: [Python-Dev] Re: ConfigParser changes
In-Reply-To: <20021231183102.GA23564@thyrsus.com>
References: <20021230234139.GE12063@epoch.metaslash.com>
 <20021231014355.GA14992@thyrsus.com>
 <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
 <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
 <200212311550.gBVFojY09030@odiug.zope.com>
 <15889.50803.219337.935244@grendel.zope.com>
 <20021231165701.GA22958@thyrsus.com>
 <15889.54671.243224.307879@grendel.zope.com>
 <20021231183102.GA23564@thyrsus.com>
Message-ID: <15889.59810.427468.150879@grendel.zope.com>

Eric S. Raymond writes:
 > No.  But comments are fair game to be discarded, since they're not part
 > of the data content.  In my use case they're not important.

I don't think they're fair game if we want a surgical implementation.
Neither are blank lines.

 > > If we're going to add a special syntax for multi-line values, we may
 > > need to think about encoding special values, text encoding, and
 > > Unicode.  And that's never easy to get concensus on.  ;-)
 > 
 > Then we shouldn't try.  Let's not let the ideal be the enemy of the good
 > here; the multiline-literal syntax is othorgonal to how we handle 
 > string encoding, and internationalization can be layered on it later.

The question of exactly where to draw the line has bit us so many
times with ConfigParser that I'm quite wary.  I'll leave it for others
to determine whether I'm too much so.

 > > I think "suggests" is too strong a word; I was only citing precedence,
 > > not advocating that approach.  I think it's pretty ugly as well.
 > 
 > I'm taking suggestions on how to do it right.  I don't think there's any
 > obviously better way than what I've implemented, except maybe for making the
 > trigger a real method rather than a magic member.

I'm not suggesting that there's a better way, only a precedent.

 > I had this conversation with Guido a couple years back.  I too would have
 > been happier with two classes, one read-only and one that supports writing
 > back out.  Here is whatt Guido said and I replied:

[long quotation elided]

 > Guido dropped the thread, which I took to mean that he saw the force
 > of the objection.  But I'd *still* like to do as he suggested.  
 > 
 > If the consensus is that its OK to deprecate add_section() and write()
 > in the base class, I'll write ConfigEditor and move all my surgery
 > stuff over there in a heartbeat.

I think add_section() is useful for modifying an in-memory
configuration (perhaps loaded from multiple sources, other than INI
files).  The implementation would be different than for your proposed
ConfigEditor, but that's OK.  write() may also be useful for other
reasons, and probably isn't worth deprecating.

Maintenance of those methods is not a good reason to create a separate
ConfigEditor class.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation


From ark@research.att.com  Tue Dec 31 19:21:45 2002
From: ark@research.att.com (Andrew Koenig)
Date: 31 Dec 2002 14:21:45 -0500
Subject: [Python-Dev] PEP 303: Extend divmod() for Multiple Divisors
In-Reply-To: <20021231184457.GE31681@laranja.org>
References: <200212311738.SAA02244@proton.lysator.liu.se>
 <20021231184457.GE31681@laranja.org>
Message-ID: <yu99adimvvsm.fsf@europa.research.att.com>

Lalo> I'm +0 on this (reasons below), but since Guido already said no:
Lalo> I find this function pretty useful, as it serves to collapse
Lalo> ugly code, and I hate ugly python code.  However, the name
Lalo> sucks.  With your changes, 'divmod' doesn't describe it anymore.

Lalo> So I suggest you come up with a better name and offer it as a
Lalo> contribution to, say, the math module.  In that case I would
Lalo> prefer to have the reverse function too.

If it helps, APL uses the names "base" and "represent" for these
two functions.

-- 
Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark


From esr@thyrsus.com  Tue Dec 31 19:23:32 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 14:23:32 -0500
Subject: [Python-Dev] Re: ConfigParser changes
In-Reply-To: <15889.59810.427468.150879@grendel.zope.com>
References: <20021231121903.GK12063@epoch.metaslash.com> <20021231145639.GA22240@thyrsus.com> <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> <20021231183102.GA23564@thyrsus.com> <15889.59810.427468.150879@grendel.zope.com>
Message-ID: <20021231192332.GA23881@thyrsus.com>

Fred L. Drake, Jr. <fdrake@acm.org>:
> I don't think they're fair game if we want a surgical implementation.
> Neither are blank lines.

That depends on whether the surgery is at the data level or the 
representation level.  The former covers my use case.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From fdrake@acm.org  Tue Dec 31 19:35:33 2002
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 31 Dec 2002 14:35:33 -0500
Subject: [Python-Dev] Re: ConfigParser changes
In-Reply-To: <20021231192332.GA23881@thyrsus.com>
References: <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
 <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
 <200212311550.gBVFojY09030@odiug.zope.com>
 <15889.50803.219337.935244@grendel.zope.com>
 <20021231165701.GA22958@thyrsus.com>
 <15889.54671.243224.307879@grendel.zope.com>
 <20021231183102.GA23564@thyrsus.com>
 <15889.59810.427468.150879@grendel.zope.com>
 <20021231192332.GA23881@thyrsus.com>
Message-ID: <15889.61829.425684.299710@grendel.zope.com>

 >Fred L. Drake, Jr. <fdrake@acm.org>:
 > > I don't think they're fair game if we want a surgical implementation.
 > > Neither are blank lines.

Eric S. Raymond writes:
 > 
 > That depends on whether the surgery is at the data level or the 
 > representation level.  The former covers my use case.



  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation


From tim@multitalents.net  Tue Dec 31 19:35:48 2002
From: tim@multitalents.net (Tim Rice)
Date: Tue, 31 Dec 2002 11:35:48 -0800 (PST)
Subject: [Python-Dev] Patch 640843 - split OPT into two parts
In-Reply-To: <15889.52760.92971.951623@montanaro.dyndns.org>
References: <15889.52760.92971.951623@montanaro.dyndns.org>
Message-ID: <Pine.UW2.4.51.0212311128530.5296@ou8.int.multitalents.net>

On Tue, 31 Dec 2002, Skip Montanaro wrote:

> There are some structural changes in configure.in that make it worth a
> careful look-see.  In particular, I hope I haven't bolluxed up Martin's SCO
> changes from September (I can't test on that platform).

I wouldn't worry about breaking SCO. It's been broken since the
decision to define _XOPEN_SOURCE.

-- 
Tim Rice				Multitalents	(707) 887-1469
tim@multitalents.net



From fdrake@acm.org  Tue Dec 31 19:37:34 2002
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Tue, 31 Dec 2002 14:37:34 -0500
Subject: [Python-Dev] Re: ConfigParser changes
In-Reply-To: <20021231192332.GA23881@thyrsus.com>
References: <20021231121903.GK12063@epoch.metaslash.com>
 <20021231145639.GA22240@thyrsus.com>
 <200212311522.gBVFMIY14143@odiug.zope.com>
 <20021231153538.GA22602@thyrsus.com>
 <200212311550.gBVFojY09030@odiug.zope.com>
 <15889.50803.219337.935244@grendel.zope.com>
 <20021231165701.GA22958@thyrsus.com>
 <15889.54671.243224.307879@grendel.zope.com>
 <20021231183102.GA23564@thyrsus.com>
 <15889.59810.427468.150879@grendel.zope.com>
 <20021231192332.GA23881@thyrsus.com>
Message-ID: <15889.61950.238264.559700@grendel.zope.com>

(Sorry for the empty reply; fingers were *way* too quick!)

I wrote:
 > I don't think they're fair game if we want a surgical implementation.
 > Neither are blank lines.

Eric S. Raymond writes:
 > That depends on whether the surgery is at the data level or the 
 > representation level.  The former covers my use case.

At the abstract data level, the order is fair game as well.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Zope Corporation


From martin@v.loewis.de  Tue Dec 31 19:38:15 2002
From: martin@v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 31 Dec 2002 20:38:15 +0100
Subject: [Python-Dev] sstruct
In-Reply-To: <20021231183117.GD31681@laranja.org>
References: <3E11C609.5000206@v.loewis.de> <r01050400-1022-F262B8D41CE711D7A7C4003065D5E7E4@[10.0.0.3]> <20021231183117.GD31681@laranja.org>
Message-ID: <3E11F227.9060904@v.loewis.de>

>>>On the latter point: What application is finding it so useful that
>>>they incorporate it in their distribution?
> 
> 
> Heck, I would have used it instead of struct in all places I ever used
> struct, if I knew about it before :-)

Can you explain why? Isn't it much slower than struct?

Regards,
Martin




From tim@zope.com  Tue Dec 31 19:40:06 2002
From: tim@zope.com (Tim Peters)
Date: Tue, 31 Dec 2002 14:40:06 -0500
Subject: [Python-Dev] New leak
Message-ID: <BIEJKCLHCIOIHAGOKOLHKEAJEGAA.tim@zope.com>

"A leak" in the datetime test turned out to be in the scaffolding that
prints the current refcount.  Here's a whittled test:

import sys
lastrc = 0
while True:
    thisrc = sys.gettotalrefcount()
    if 1:   # change 1 to 0 and there's no leak
        x = '*' * 10  # This leaks; ditto tuple times int, or list times int
    print >> sys.stderr, 'total refs:', thisrc, 'delta:', thisrc - lastrc
    lastrc = thisrc

Run that in a debug build, and it leaks one reference per loop trip.



From esr@thyrsus.com  Tue Dec 31 20:10:38 2002
From: esr@thyrsus.com (Eric S. Raymond)
Date: Tue, 31 Dec 2002 15:10:38 -0500
Subject: [Python-Dev] Re: ConfigParser changes
In-Reply-To: <15889.61950.238264.559700@grendel.zope.com>
References: <200212311522.gBVFMIY14143@odiug.zope.com> <20021231153538.GA22602@thyrsus.com> <200212311550.gBVFojY09030@odiug.zope.com> <15889.50803.219337.935244@grendel.zope.com> <20021231165701.GA22958@thyrsus.com> <15889.54671.243224.307879@grendel.zope.com> <20021231183102.GA23564@thyrsus.com> <15889.59810.427468.150879@grendel.zope.com> <20021231192332.GA23881@thyrsus.com> <15889.61950.238264.559700@grendel.zope.com>
Message-ID: <20021231201038.GA24063@thyrsus.com>

Fred L. Drake, Jr. <fdrake@acm.org>:
> Eric S. Raymond writes:
>  > That depends on whether the surgery is at the data level or the 
>  > representation level.  The former covers my use case.
> 
> At the abstract data level, the order is fair game as well.

My use case calls that assummption into some question.  There
are attribute groups within sections where I think the meaning
is order-dependent.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>


From lists@morpheus.demon.co.uk  Tue Dec 31 20:24:45 2002
From: lists@morpheus.demon.co.uk (Paul Moore)
Date: Tue, 31 Dec 2002 20:24:45 +0000
Subject: [Python-Dev] Write All New Import Hooks (PEP 302) in Python,
 Not C
References: <3E106BBB.4020909@interet.com> <r01050400-1022-8AC2815C1CC911D7A7C4003065D5E7E4@[10.0.0.3]>
Message-ID: <n2m-g.lm26hr76.fsf@morpheus.demon.co.uk>

Just van Rossum <just@letterror.com> writes:

> This is all only half true. sys.path_importer_cache is exposed so
> you can *clear* it in case you install a path hook that should take
> over sys.path items that may already have a handler. To only
> partially clear it is advanced usage. By no means you're supposed to
> *add* to sys.path_importer_cache explicitly, as someone after you
> may clear it again, and then you're hosed if there's no proper path
> hook.

Whoo. I was thinking about how I'd simulate the (now unavailable)
feature of adding importers directly to sys.path. It took me virtually
no time at all to come up with

    importer = MyImporter(...)
    imp_id = str(id(importer)) # A cookie
    sys.path.append(imp_id)
    sys.path_importer_cache[imp_id] = importer

The only way I could come up with of doing the same thing without
messing with sys.path_importer_cache was by implementing my own
equivalent - basically

    id_cache = {}
    def id_factory(id):
        try:
            return id_cache[id]
        except KeyError:
            raise ImportError
    sys.path_hooks.append(id_factory)

    importer = MyImporter(...)
    imp_id = str(id(importer)) # A cookie
    sys.path.append(imp_id)
    id_cache[imp_id] = importer

I can't think of any reason why this is *intrinsically* better than
the first way other than because someone may clear the cache on
you. And it's certainly not as intuitive.

> I also disagree that adding an importer to sys.path_importer_cache
> would always be *easier* than doing it the right way by adding a
> hook to sys.path_hooks.

Hmm. See above. OK, so that's generic code, and specific cases may
always be easier to code "the right way".

> If would be somewhat easier for "cookie" path items, but I think in
> 99% of these cases a path-less importer on sys.meta_path would work
> just as well.

But sys.meta_path happens *before* sys.path. It's entirely likely that
a cookie item might want to run *after* sys.path. For that we need
"stage 2", where the real sys.path importer is a proper item on
sys.meta_path.

At the moment, whether sys.meta_path should run before or after
sys.path is a judgement call - which usage is more common? In the
longer term, the normal insert(0) vs append() approach gives both
options.

Disclaimer: As usual, this is all theory. Real use cases take
precedence.

Paul.

PS Jim's comments have now raised two (as far as I can recall) issues
   with the proposed hook mechanism which should be documented in the
   "Open Issues" section of the PEP:

   1. sys.meta_path allows hooks to be added *before* sys.path
      processing, but there is no equivalent facility for hooks
      *after* sys.path (response: either wait for "phase 2" or use a
      cookie at the end of sys.path).
   2. There is no easy way of "stacking" hooks (response: there isn't
      one now, either, so that's not a critical issue).
-- 
This signature intentionally left blank


From lalo@laranja.org  Tue Dec 31 21:11:48 2002
From: lalo@laranja.org (Lalo Martins)
Date: Tue, 31 Dec 2002 19:11:48 -0200
Subject: [Python-Dev] sstruct
In-Reply-To: <3E11F227.9060904@v.loewis.de>
References: <3E11C609.5000206@v.loewis.de> <r01050400-1022-F262B8D41CE711D7A7C4003065D5E7E4@[10.0.0.3]> <20021231183117.GD31681@laranja.org> <3E11F227.9060904@v.loewis.de>
Message-ID: <20021231211148.GL31681@laranja.org>

On Tue, Dec 31, 2002 at 08:38:15PM +0100, "Martin v. Löwis" wrote:
> >Heck, I would have used it instead of struct in all places I ever used
> >struct, if I knew about it before :-)
> 
> Can you explain why? Isn't it much slower than struct?

I haven't yet tested its speed, but Just claims it's just a bit slower.

However, in a choice between clarity and speed I almost always, scratch
that, always choose clarity.  Then *if* I have performance problems I go
back and speed up where I can, as long as I don't compromise clarity beyond
some reasonable arbitrary limit.  The struct module, however, falls a lot
beyond this limit, and if I had a choice I wouldn't even have used it at all.

(Short answer: Mu.)

[]s,
                                               |alo
                                               +----
--
            Those who trade freedom for security
               lose both and deserve neither.
--
http://www.laranja.org/                mailto:lalo@laranja.org
         pgp key: http://www.laranja.org/pessoal/pgp

Eu jogo RPG! (I play RPG)         http://www.eujogorpg.com.br/
GNU: never give up freedom                 http://www.gnu.org/


From tim.one@comcast.net  Tue Dec 31 22:14:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 31 Dec 2002 17:14:17 -0500
Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Modules _randommodule.c,1.1,1.2
In-Reply-To: <E18TUM3-0003L6-00@sc8-pr-cvs1.sourceforge.net>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEECAEGAA.tim.one@comcast.net>

[nnorwitz@users.sourceforge.net]
> Modified Files:
> 	_randommodule.c
> Log Message:
> Try to get compilation working for cygwin
>
> Index: _randommodule.c

And then, for example, before:

> ! 	PyObject_GenericGetAttr,	/*tp_getattro*/

and after:

> ! 	0,				/*tp_getattro*/

followed by:

> + 	Random_Type.tp_getattro = PyObject_GenericGetAttr;
> + 	Random_Type.tp_alloc = PyType_GenericAlloc;
> + 	Random_Type.tp_free = _PyObject_Del;

in the module init function.

Please don't make this kind of change -- it makes the code so much harder to
follow.  If this is needed for Cygwin, then, e.g., do

#define DEFERRED(x) 0 /* some boxes can't resolve addresses at compile-time
*/

and make the "after" line

     DEFERRED(PyObject_GenericGetAttr), /*tp_getattro*/

IOW, the type slots should be readable on their own, as a static unit.



From neal@metaslash.com  Tue Dec 31 22:28:23 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 31 Dec 2002 17:28:23 -0500
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Modules _randommodule.c,1.1,1.2
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHEECAEGAA.tim.one@comcast.net>
References: <E18TUM3-0003L6-00@sc8-pr-cvs1.sourceforge.net> <BIEJKCLHCIOIHAGOKOLHEECAEGAA.tim.one@comcast.net>
Message-ID: <20021231222823.GB20404@epoch.metaslash.com>

On Tue, Dec 31, 2002 at 05:14:17PM -0500, Tim Peters wrote:
> 
> And then, for example, before:
> 
> > ! 	PyObject_GenericGetAttr,	/*tp_getattro*/
> 
> and after:
> 
> > ! 	0,				/*tp_getattro*/
> 
> followed by:
> 
> > + 	Random_Type.tp_getattro = PyObject_GenericGetAttr;
> > + 	Random_Type.tp_alloc = PyType_GenericAlloc;
> > + 	Random_Type.tp_free = _PyObject_Del;
> 
> in the module init function.
> 
> Please don't make this kind of change -- it makes the code so much harder to
> follow.  If this is needed for Cygwin, then, e.g., do
> 
> #define DEFERRED(x) 0 /* some boxes can't resolve addresses at compile-time
> */
> 
> and make the "after" line
> 
>      DEFERRED(PyObject_GenericGetAttr), /*tp_getattro*/
> 
> IOW, the type slots should be readable on their own, as a static unit.

I agree, this makes sense.  The only problem is that the
functions/methods will have to be specified twice.  It would be nice
if we only had to specify each once.  Any ideas?

There are several places this has already been done (and at least one
more coming).  Do you want a global Py_DEFERRED() or some other name?

Neal




From guido@python.org  Tue Dec 31 23:35:07 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 31 Dec 2002 18:35:07 -0500
Subject: [Python-Dev] Python 2.3a1 released
Message-ID: <200212312335.gBVNZ7p15291@pcp02138704pcs.reston01.va.comcast.net>

I'm happy to announce the successful release of Python 2.3a1 tonight,
on Dec 31 (in *some* timezones :-).  Go pick it up from its home page:

  http://www.python.org/2.3/

This is an alpha release: if you have Python code that's important (to
you), please test it thoroughly with this release, but don't use the
release for production runs.  Please report any problems to the
SourceForge bug tracker:

  http://sourceforge.net/bugs/?group_id=5470

What's new?  The language isn't changing much this time.  Instead, we
have lots of new or improved standard library modules: bsddb, bz2,
datetime, heapq, logging, optparse, ossaudiodev, random (Mersenne
Twister algorithm), sets, socket (added timeouts), textwrap, and
zipimport.  That's right, you can now import modules from zip files.
More elaborate news is on the website:

  http://www.python.org/2.3/highlights.html	(brief)
  http://www.python.org/doc/2.3a1/whatsnew/	(extensive)
  http://www.python.org/2.3/NEWS.html		(exhausting)

--Guido van Rossum (home page: http://www.python.org/~guido/)


From dave@boost-consulting.com  Mon Dec 30 16:40:58 2002
From: dave@boost-consulting.com (David Abrahams)
Date: Mon, 30 Dec 2002 11:40:58 -0500
Subject: [Python-Dev] Extension modules, Threading, and the GIL
References: <ufzsgi9ao.fsf@boost-consulting.com> <20021230145912.GA8481@panix.com>
 <ud6njfshh.fsf@boost-consulting.com>Gcc: nnfolder+archive:sent
Message-ID: <ulm27cvdx.fsf@boost-consulting.com>

The following message is a courtesy copy of an article
that has been posted to comp.lang.python as well.


David Abrahams <dave@boost-consulting.com> writes:

> Aahz <aahz@pythoncraft.com> writes:
>> I think this thread might be better handled on c.l.py, at least
>> until it's understood well enough to be clear whether something does
>> need to change in Python.
>
> I'm pretty certain I understand what's going on.  If it would be
> better for you to take this to c.l.py, I'm happy to do so, but AFAICT
> there _is_ a Python core issue here: there's no way to find out
> whether you've already got the GIL**, so if you _might_ have to acquire
> it, you must always acquire it.
>
> **If I'm wrong about that, please let me know.  It isn't obvious from
>   the documentation.

This is the continuation of a thread originated on python-dev; I was
asked to re-raise it here on python-list until the issue is better
understood.

The original posting was here:

  http://aspn.activestate.com/ASPN/Mail/Message/python-dev/1482879

The he essence of the problem is that there's no way to write code
that uses the Python 'C' API and which has no knowledge of whether it
is running on Python's main thread when it is entered.

The two respondents were left with some questions; you can read those,
and my responses, in the thread at the bottom of the page referenced
above.

Thanks for your attention,
Dave

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution