Mailman 3 December 2002 - Python-Dev

Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1
by Michael Hudson Dec. 3, 2002

Dec. 3, 2002

tim_one(a)users.sourceforge.net writes: > New program just to display pickle sizes. This makes clear that the > copy_reg based C implementation is much more space-efficient in the > end than the __getstate__/__setstate__ based Python implementation, > but that 4-byte date objects still suffer > 10 bytes of overhead each > no matter how many of them you pickle in one gulp. Presumably there's a possibility of an optimization for pickling homogeneous (i.e. all the same type) … [View More]

2 1

Re: [Python-Dev] Re: [Python-checkins] python/nondist/sandbox/datetime picklesize.py,NONE,1.1
by Michael Hudson Dec. 3, 2002

Dec. 3, 2002

Michael Hudson <mwh(a)python.net> writes: > tim_one(a)users.sourceforge.net writes: > > > New program just to display pickle sizes. This makes clear that the > > copy_reg based C implementation is much more space-efficient in the > > end than the __getstate__/__setstate__ based Python implementation, > > but that 4-byte date objects still suffer > 10 bytes of overhead each > > no matter how many of them you pickle in one gulp. > > … [View More]Presumably there's a possibility of an optimization for pickling > homogeneous (i.e. all the same type) lists (in pickle.py, not here). > > Hard to say whether it would be worth it, though. Here's a fairly simple minded patch to the pickling side of pickle.py: it seems to save about 6 bytes per object in the good cases. with: list of 100 dates via C -- 1236 bytes, 12.36 bytes/obj without: list of 100 dates via C -- 1871 bytes, 18.71 bytes/obj I'm not going to pursue this further unless someone thinks it's a worthwhile move. Cheers, M. Index: pickle.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/pickle.py,v retrieving revision 1.72 diff -c -r1.72 pickle.py *** pickle.py 13 Nov 2002 22:01:26 -0000 1.72 --- pickle.py 3 Dec 2002 18:24:37 -0000 *************** *** 109,114 **** --- 109,115 ---- INST = 'i' LONG_BINGET = 'j' LIST = 'l' + HOM_LIST = 'k' EMPTY_LIST = ']' OBJ = 'o' PUT = 'p' *************** *** 439,445 **** --- 440,478 ---- def save_empty_tuple(self, object): self.write(EMPTY_TUPLE) + def save_hom_list(self, object): + reduce = dispatch_table[type(object[0])] + + write = self.write + save = self.save + memo = self.memo + + write(HOM_LIST) + + o = object[0] + + c, a = reduce(o) + + l = [a] + + for o in object[1:]: + l.append(reduce(o)[1]) + + save(c) + save(l) + def save_list(self, object): + t = {} + + for o in object: + t[type(o)] = 1 + if len(t) > 1: + break + else: + if t and dispatch_table.has_key(t.iterkeys().next()): + self.save_hom_list(object) + return + d = id(object) write = self.write -- Unfortunately, nigh the whole world is now duped into thinking that silly fill-in forms on web pages is the way to do user interfaces. -- Erik Naggum, comp.lang.lisp [View Less]

1 0

Re: [Python-Dev] New and Improved Import Hooks
by Moore, Paul Dec. 3, 2002

Dec. 3, 2002

(I'm reading python-dev via the web, so this is a combined reply to various posts) First of all, let me say that I'm interested in imports from zipfiles, as a feature. I don't have any intrinsic reason to prefer a specific implementation over any other. I picked up Jim Ahlstrom's patch and had a look at it simply because it seemed the best way to help. My view is that import.c is, indeed, very complex and difficult code to follow. I haven't made a significant attempt to do so, taking the view … [View More]

4 3

Re: [Python-Dev] Doc/whatsnew/* v. Misc/NEWS
by Michael Hudson Dec. 3, 2002

Dec. 3, 2002

Skip Montanaro <skip(a)pobox.com> writes: > Do we really need both Doc/whatsnew/* and Misc/NEWS? It seems that people > frequently make nearly the same edits to both files, or add to Misc/NEWS > only and someone later (usually Andrew) goes back and twiddles the necessary > bits in Doc/whatsnew. I'd have said yes: Misc/NEWS is for "developers" and whatsnew is for "users", although this is blatantly a bogus categorization. NEWS items are brief to the point of curtness, … [View More]

2 1

RE: [Python-Dev] New and Improved Import Hooks
by Moore, Paul Dec. 3, 2002

Dec. 3, 2002

From: Just van Rossum [mailto:just@letterror.com] > > I've explained this a couple of times now. Please read > > the PEP for details, but Jim claims that zipfile imports > > need this if they are not to be too slow. > Well, I am not going to need it: upon reading the zip > contents table I build a dict, mapping (full) module > names to toc info. One dict lookup will be enough to > get at the right data. But that's exactly what Jim's cache is doing. His code is … [View More]

2 1

New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
by Wiktor Sadowski Dec. 3, 2002

Dec. 3, 2002

I think Python should have a more universal import mechanism , which could be done by adding a new C_CUSTOM type to the definitions for dynamic loading (importdl.h) , a new struct _customtab to import.h and a new function to Python API (import.c). Then any archive,database,internet import architecture could be implemented in C extensions without messing with Python core. Wiktor Sadowski www.wiktorsadowski.com IMPLEMENTATION (tested on win_98) import.h extern DL_IMPORT(int) … [View More]PyImport_AppendCustomtab(int (*get_name)(char*),void (*initfunc)(char*)); struct _customtab { int (*get_name)(char*); void (*initfunc)(char*); struct _customtab *next; }; importdl.h enum filetype { SEARCH_ERROR, PY_SOURCE, PY_COMPILED, C_EXTENSION, PY_RESOURCE, /* Mac only */ PKG_DIRECTORY, C_BUILTIN, PY_FROZEN, C_CUSTOM, PY_CODERESOURCE /* Mac only */ }; import.c struct _customtab *PyImport_Customtab; int PyImport_AppendCustomtab(int (*get_name)(char*),void (*initfunc)(char*)) { struct _customtab *pycustomtab; struct _customtab *newtab=PyMem_NEW(struct _customtab,1); newtab->get_name=get_name; newtab->initfunc=initfunc; newtab->next=NULL; pycustomtab=PyImport_Customtab; if (pycustomtab) { while(pycustomtab->next) pycustomtab=pycustomtab->next; pycustomtab->next=newtab; } else PyImport_Customtab=newtab; return 1; } static struct filedescr * find_module(char *realname, PyObject *path, char *buf, size_t buflen, FILE **p_fp) ................................................................................................... static struct filedescr fd_custom = {"", "", C_CUSTOM}; ................................................................................................... before find_frozen !: if (is_custom(name)) { strcpy(buf, name); return &fd_custom; } if (find_frozen(name) != NULL) { strcpy(buf, name); return &fd_frozen; ................................................................................................... and again: if (path == NULL) { if (is_custom(name)) { strcpy(buf, name); return &fd_custom; } LOAD_MODULE static PyObject * load_module(char *name, FILE *fp, char *buf, int type) ................................................................................................... switch (type) { ............................................. case C_CUSTOM: if (buf != NULL && buf[0] != '\0') name = buf; err = init_custom(name); if (err < 0) return NULL; if (err == 0) { PyErr_Format(PyExc_ImportError, "Purported %s module %.200s not found", type == C_BUILTIN ? "builtin" : type == C_CUSTOM ? "custom":"frozen", name); return NULL; } modules = PyImport_GetModuleDict(); m = PyDict_GetItemString(modules, name); if (m == NULL) { PyErr_Format( PyExc_ImportError, "%s module %.200s not properly initialized", type == C_BUILTIN ? "builtin" : type == C_CUSTOM ? "custom":"frozen", name); return NULL; } Py_INCREF(m); break; IS_CUSTOM static int is_custom(char *name) { struct _customtab *p; if (!PyImport_Customtab) return 0; p=PyImport_Customtab; while(p){ if (p->get_name) if ((*p->get_name)(name)) return 1; p=p->next; } return 0; } INIT_CUSTOM static int init_custom(char *name) { struct _customtab *p = PyImport_Customtab; if (_PyImport_FindExtension(name, name) != NULL) return 1; while(p) { if (p->get_name){ if ((*p->get_name)(name)) { if (p->initfunc == NULL) { PyErr_Format(PyExc_ImportError, "Cannot re-init internal module %.200s", name); return -1; } if (Py_VerboseFlag) PySys_WriteStderr("import %s # builtin\n", name); (*p->initfunc)(name); if (PyErr_Occurred()) return -1; if (_PyImport_FixupExtension(name, name) == NULL) return -1; return 1; } } p=p->next; } return 0; } [View Less]

7 14

Re: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
by Michael Hudson Dec. 3, 2002

Dec. 3, 2002

David Abrahams <dave(a)boost-consulting.com> writes: > My users want extension modules to be able to control how their > attributes are imported when using "import *". In particular, > Boost.Python includes a kind of multimethod facility, and some of my > users would like it if > > from bpl_extension import * > > caused multimethod definitions in bpl_extension to be /merged/ into > the current module, rather than replacing any multimethod bindings > … [View More]

2 1

Re: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
by Just van Rossum Dec. 3, 2002

Dec. 3, 2002

Greg Ward wrote: > > - a hook either returns None ("module not found") or a two-tuple: > > (loadfunc, cookie). loadfunc is a callable object that will > > Sounds sensible, but please s/cookie/data/ (or similar). Ok. > > be called with two arguments: the fullname and "cookie". The > > cookie is just an arbitrary object, private to the hook. The > > loaderfunc must return the imported module. > > - (Currently it expects … [View More]

6 10

Re: [Python-Dev] extension module .o files wind up in the wrong place
by Michael Hudson Dec. 3, 2002

Dec. 3, 2002

Skip Montanaro <skip(a)pobox.com> writes: > I don't have time to look into this right now. I only just noticed it while > I was trying to do parallel builds in build.dbg and build.opt directories. > The debug .o files got written first, so when the optimized make came along, > there were already .o files, which had been compiled with --with-pydebug, > got linked improperly. Ah, this would be why I couldn't build a debug python on my iBook, then. If this is Jack's fault, … [View More]

1 0

Re: [Python-Dev] New universal import mechanism ( Re: [Python-checkins] python/dist/src/Python import.c,2.210,2.211)
by Wiktor Sadowski Dec. 3, 2002

Dec. 3, 2002

Just van Rossum wrote: >>Could you please post a normal diff of your patch? It looks very interesting. Sorry, I haven't got time for cleaning up my modified Python C code and building the diffs. >.I'm not entirely sure yet how my idea and yours could be integrated, but I think > it's work a look. The concept is the same , there are differences in implementations. The implementation you have posted is much more pure Python oriented. I am using C structures and pointers to C … [View More]

2 1