Mailman 3 December 2002 - Python-Dev

Re: [Python-Dev] zipimport, round 3 (or would that be that 37?)
by Wiktor Sadowski Dec. 8, 2002

Dec. 8, 2002

Just van Rossum <just(a)letterror.com> writes: > I'll write a bit about the mechanics tomorrow, but in the meantime I would like > to see some test results/bug reports/bug fixes from Windows developers. Please > don't start with the full monty (ie. the std lib), try a small test archive > first please ;-) I have managed to apply the patch (import.c) to Python2.2.2 (wasn't too hard) Could you please email me the site.py patch,or post it again? >>>base64.decodestring(site.patch) >>>binascii.Error: Incorrect padding Python 2.2.2 (#37, Dec 7 2002, 19:19:43) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. >>> import sys >>> print sys.import_hooks [<type 'zipimport.zipimporter'>] >>> from PIL import Image Traceback (most recent call last): File "<pyshell#2>", line 1, in ? from PIL import Image ImportError: No module named PIL >>> sys.path.append('C:\Python22\PIL.zip') >>> from PIL import Image now it works >>> from zipimport import zipimporter >>> arc=zipimporter('C:\Python22\Modules.zip') #Modules.zip - zipped non-package directory >>> m=arc.load_module('module1') ok >>> arc1=zipimporter('C:\Python22\Modules.zip\Folder1') >>> m=arc1.load_module('fmodule1') ok Regards Wiktor art(a)wiktorsadowski.com

1 0

Weekly Python Bug/Patch Summary
by Skip Montanaro Dec. 8, 2002

Dec. 8, 2002

Bug/Patch Summary ----------------- 316 open / 3105 total bugs (+2) 94 open / 1822 total patches (-2) New Bugs -------- ext module generation problem (2002-08-23) http://python.org/sf/599248 Tkinter: BitmapImage vanishes if not stored in non-local var (2002-11-01) http://python.org/sf/632323 test_poll fails on FreeBSD (2002-12-01) http://python.org/sf/646547 netrc & special chars in passwords (2002-12-01) http://python.org/sf/646592 Can't run script from IDLE (2002-12-01) http://python.org/sf/646730 more email ASCII decoding errors (2002-12-03) http://python.org/sf/648119 xmlrpc can't do proxied HTTP (2002-12-04) http://python.org/sf/648658 nturl2path.url2pathname() mishandles /// (2002-12-07) http://python.org/sf/649961 All URL funcs mishandle file://localhost (2002-12-07) http://python.org/sf/649962 urllib.url2pathname('file://host/')... (2002-12-07) http://python.org/sf/649974 New Patches ----------- asyncore loop_once and remove poll2 (2002-12-04) http://python.org/sf/648322 tuple arg for issubclass (2002-12-06) http://python.org/sf/649608 urllib2.Request's headers are case-sens. (2002-12-06) http://python.org/sf/649742 Fix: asynchat.py: endless loop (2002-12-06) http://python.org/sf/649762 Complementary patch for OpenVMS (2002-12-07) http://python.org/sf/649997 posixfy some things (2002-12-08) http://python.org/sf/650412 avoid redefinition of pthread functions (2002-12-08) http://python.org/sf/650415 Closed Bugs ----------- Removing _tkinter considered harmful (2002-03-25) http://python.org/sf/534748 LINKCC incorrectly set (2002-06-16) http://python.org/sf/569668 list slice ass ignores subtypes of list (2002-11-04) http://python.org/sf/633152 --with-dl-dld section of the readme (2002-11-10) http://python.org/sf/636313 __all__ not in RefMan index (2002-11-24) http://python.org/sf/643227 typo in Object/abstract.c : ternary_op (2002-11-24) http://python.org/sf/643260 old UnicodeData.txt (2002-12-01) http://python.org/sf/646408 Closed Patches -------------- Start of a bgen tutorial (2001-09-04) http://python.org/sf/458383 use readline in pydoc.help if available (2002-04-28) http://python.org/sf/549901 Regression exceptions for cygwin (2002-05-03) http://python.org/sf/551977 PyBufferObject fixes (2002-05-04) http://python.org/sf/552438 db4 include not found (2002-08-02) http://python.org/sf/590377 OpenVMS patches (2002-09-24) http://python.org/sf/614055 make some type attrs writable (2002-11-09) http://python.org/sf/635933 Allow device name in linuxaudiodev.open (2002-11-27) http://python.org/sf/644977 Give some clue why modules fail... (2002-11-28) http://python.org/sf/645383

1 0

dummy_thread (#622537), part 2
by Brett Cannon Dec. 8, 2002

Dec. 8, 2002

Wrote ``dummy_threading``. Thanks to the recent importing discussion and one of Martin's replies to my previous announcement for this patch I figured out how to do it while still using ``threading``. ``sys.modules`` is your friend (as long as you treat it nicely). So the patch is ready for review again. Tim mentioned he was going to have a look, but I will let him say whether he is going to still look at it now. All relevent info is in the patch. -Brett

1 0

Re: [Python-Dev] os.path.commonprefix
by Michael Hudson Dec. 7, 2002

Dec. 7, 2002

Armin Rigo <arigo(a)tunes.org> writes: > >>> commonprefix(['/home/users/arigo', '/home/users/alal']) > '/home/users/a' > > What is the use for such a thing ?? I seem to recall a discussion which decided that there is no use for such a thing, but that changing it might break code for people who found one. ICBW. > I suggest that a more descriptive comment be added in the documentation. No argument from here. Cheers, M. -- We did requirements and task analysis, iterative design, and user testing. You'd almost think programming languages were an interface between people and computers. -- Steven Pemberton (one of the designers of Python's direct ancestor ABC)

2 1

zipimport & import hooks
by Just van Rossum Dec. 7, 2002

Dec. 7, 2002

I think we can separate the zipimport issue from the import hooks issue, but only to an extent. *Some* hook need to be installed in import.c for zipimports to work, but it doesn't neccesarily mean we must add a general new import hook scheme *now*. But then again, it might be a *very* small step from enabling zipimports to exposing the needed hook and have a clean and general solution. It seems the majority doesn't like non-strings on sys.path, which to me is understandable, especially if you take PYTHONPATH into consideration. Let's try to focus on that. Here's a minimal implementation idea that will allow .zip filenames on sys.path: zipimporters = {} # a cache, private to import.c for p in sys.path: if p.endswith(".zip"): z = zipimporters.get(p) if z is None: z = zipimporters[p] = zipimporter(p) loader = z.find_module(name) ...etc... else: ...builtin import... This gets nessesarily more complex for packages, as now we also have to cater for __path__ items of this form: "/path/to/my/archive.zip/packagedir". In this case we need to create a "subimporter" of "/path/to/my/archive.zip" if only because we don't want to reread the zip archive's file index. By now the machinery needed can be written like so: zipimporters = {} # a cache, private to import.c def get_zipimporter(p): if p.endswith(".zip"): return zipimporter(p) if not os.path.exists(p): pos = p.rfind(".zip") if pos > 0: archive = p[:pos + 4] subpath = p[pos + 5:] # skip initial sep z = zipimporters[archive] return z.subimporter(subpath) for p in sys.path: importer = zipimporters.get(p, -1) if importer != -1: importer = zipimporters[p] = get_importer(p) if importer is not None: loader = importer.find_module(name) ...etc... else: ...builtin import... And *this* can easily be rewritten in a slightly more general form: path_importers = {} # exposed in sys import_hooks = [get_zimpimporter] # exposed in sys # get_zimpimporter() implementation left out for brevity def get_importer(p): for hook in import_hooks: importer = hook(path) if importer is not None: return None return None for p in sys.path: importer = path_importers.get(p, -1) if importer != -1: importer = path_importers[p] = get_importer(p) if importer is not None: loader = importer.find_module(name) ...etc... else: ...builtin import... Which, an observant reader might notice, is a stripped down version of Gordon's iu.py, with s/shadowpath/path_importers/ and s/ownertypes/import_hooks/ and without his metapath. (The metapath is a great idea, but I don't think it's possible to do without a *major* rewrite of import.c. The rest is relatively easy to do, in fact I've got it mostly working already.) Just

12 22

We got leaks!
by Tim Peters Dec. 7, 2002

Dec. 7, 2002

If someone is up for a good weekend puzzle, try this on for size. I'd like to multitask this, if possible. I'm attacking the datetime module in the CVS sandbox, aiming to get it into good shape for 2.3a1. There's a pure Python implementation and a pure C implementation (incomplete but already quite hefty). You build the latter like so, for release mode: setup.py build_ext -i or for debug mode: setup.py build_ext -i --debug test_both.py is a test suite. To test the Python implementation, test_both.py and to test the C implementation, test_both.py c The tests should all pass. Running this in a loop, I didn't notice any memory growth after a minute of CPU time. To be sure, I replaced test_main like so: def test_main(): import gc r = unittest.TextTestRunner(stream=sys.stdout, verbosity=2) s = test_suite() while True: r.run(s) gc.collect() print gc.garbage # this is always an empty list print '*' * 10, 'total refs:', sys.gettotalrefcount() and ran a debug build, watching the printout of total outstanding refs. Alas, the C implementation shows a very slow leak this way: ********** total refs: 29633 ********** total refs: 29643 ********** total refs: 29653 ********** total refs: 29663 ********** total refs: 29673 So the C implementation is leaking 10 references per test cycle. Fine, so I screwed up the C code. However, the Python implementation leaks close to 4x faster, 37 references per test cycle. Jeremy suggests I look for the missing decref in datetime.py <wink>: ********** total refs: 33921 ********** total refs: 33958 ********** total refs: 33995 ********** total refs: 34032 ********** total refs: 34069 Likely suspects are "newer stuff": new-style classes and class methods. Possibly unittest too, but hard to imagine how. Even if you can't make time to dig into this, it would help a little to know what the C and Python leak rates are on a different platform. Heck, it would help to know whether the C code even compiles and the tests pass on a different platform (I'm running Win2K + current CVS Python + MSVC 6 here).

5 10

PEP 242 Numeric kinds
by Paul F Dubois Dec. 6, 2002

Dec. 6, 2002

We had put off consideration of PEP 242 Numeric kinds until June 2002 when a meeting of some interested parties was to occur but the meeting didn't occur. I have a draft implementation and users of it but my feeling is that although correct and useful the PEP is not useful enough to put it in the standard library. Since it really only interests scientific programmers I propose simply making it a separate deliverable on the Numeric site and withdrawing the PEP.

7 6

RE: [Python-Dev] Another approach for the import mechanism
by Moore, Paul Dec. 6, 2002

Dec. 6, 2002

From: Martin v. Löwis [mailto:martin@v.loewis.de] > What do you mean with "was originally", and what do you mean > with "would break"? [...] Looks like I did misunderstand :-( > I recommend you try the patch, to see how it works in the real > world. At the moment the patch crashes immediately on Windows. I've let Just know, and I'll try the next version when it arrives... Paul. PS If Just's patch *doesn't* stabilise in time for 2.3, and we go back to Jim Ahlstrom's patch, I'd support Just writing a PEP to cover his import hooks mechanism, for future inclusion. (Actually, I think that Just's mechanism has sufficient extra functionality over PEP 273, that it deserves a PEP even now. But if Just takes time out to write a PEP, I suspect that makes it even less likely we'll get working code in time...)

3 2

Re: Patched Transport
by Bill Bumgarner Dec. 6, 2002

Dec. 6, 2002

On Friday, December 6, 2002, at 04:21 AM, Fredrik Lundh wrote: > hi bill, > > >> As I move to a Transport subclass that uses urllib2, I first created a >> Transport subclass whose request() method is patched to delegate all >> interaction with the connection to methods on the Transport class. > > looks fine. > > just one nit: changing the make_connection signature may break > existing programs. how about adding yet another hook: > > h = self.make_connection(host) > if verbose: > self.set_debuglevel(h, verbose) > > ... > > def set_debuglevel(self, connection, verbosity): > ... > > cheers /F I made the change as recommended -- I was being lazy... XMLRPC proxying now works and, along with it, so does ProxyHandler ordering. The fix I have does not modify urllib2 -- for my current needs, I'm trying to stick with stock python as much as possible. As such, I created a _fixUpHandlers() that will fix the handler order of any Opener instance -- likely, it should be eliminated and a similar fix integrated into add_handler() on Opener directly. I will test to make sure it works with a mix of HTTPS and with proxy servers that require authentication [I'm currently testing without a network connection on my laptop while on the train -- thankfully, OS X's apache configuration includes mod_proxy in the apache configuration... just have to turn it on]. Caveats are noted in the code. In particular, using urllib2 really wants the full URL so that it can do protocol resolution internally. I also ended up with a dummy class (_TransportConnection) that can cache-and-carry the hunks of information passed to the various send_* and get_* methods as the granularity of those methods simply does not match the calling semantics of the urllib2 library. To initialize a ServerProxy with HTTPTransport, I use the following function. Note that the arguments passed in are identifical to the initializer for ServerProxy. proxyUrl, proxyUser, proxyPass are currently stored as module variables -- this will change in the very near future, but is largely irrelevant in the context of xmlrpclib / urllib2. Of relevancy, I believe that proxyUrl could be of the form http://<user>:<pass>@host:port/path/ and it would be no different than creating a HTTPTransport by passing in the user/pass seperately. def serverProxyForUrl(uri, transport=None, encoding=None, verbose=0): if not transport: transport = HTTPTransport.HTTPTransport(uri, proxyUrl, proxyUser, proxyPass) return xmlrpclib.ServerProxy(uri, transport, encoding, verbose) (4 space indent, not tabs-- I believe this is the recommended standard for the python library?) In operation, HTTPTransport could completely replace the Transport and SafeTransport classes found within xmlrpclib. However, the one incompatible change is in the way the transport class is instantiated. The __init__ method requires a single argument; the full URI of the XML-RPC server. ServerProxy's __init__() could easily be modified to address this issue. Next, I'll probably and compressed content support -- in my app, I can reduce the bytecount sent between client and server by upwards of 90% by simply gzip'ing the XML prior to dropping it on the wire. I haven't even remotely looked at how to do this. The actually compression/decompression is trivial, but there are a lot of possible spots to stick the compression/decompression mechanism. b.bum --- """ HTTPTransport provides an urllib2 based communications channel for use in xmlrpclib. Created by Bill Bumgarner <bbum(a)mac.com>. Using HTTPTransport allows the XML-RPC library to take full advantage of the features perpetuated by urllib2, including HTTP proxy support and a number of different authentication schemes. If urllib2 does not provide a handler capable of meeting the developer's need, the developer can create a custom Handler without requiring any changes to the XML-RPC code. Example usage: def serverProxyForUrl(uri, transport=None, encoding=None, verbose=0): if not transport: transport = HTTPTransport.HTTPTransport(uri, proxyUrl, proxyUser, proxyPass) return xmlrpclib.ServerProxy(uri, transport, encoding, verbose) """ import xmlrpclib from xmlrpclib import ProtocolError from urllib import splittype, splithost import urllib2 import sys class _TransportConnection: pass def _fixHandlerArrayOrder(handlers): insertionPoint = 0 for handlerIndex in range(0, len(handlers)): aHandler = handlers[handlerIndex] if isinstance(aHandler, urllib2.ProxyHandler): del handlers[handlerIndex] handlers.insert(insertionPoint, aHandler) insertionPoint = insertionPoint + 1 def _fixUpHandlers(anOpener): ### Moves proxy handlers to the front of the handlers in anOpener # # This function preserves the order of multiple proxyhandlers, if present. # This appears to be wasted effort in that build_opener() chokes if there # is more than one instance of any given handler class in the arglist. _fixHandlerArrayOrder(anOpener.handlers) map(lambda x: _fixHandlerArrayOrder(x), anOpener.handle_open.values()) class HTTPTransport(xmlrpclib.Transport): """Handles an HTTP transaction to an XML-RPC server using urllib2 [eventually].""" def __init__(self, uri, proxyUrl=None, proxyUser=None, proxyPass=None): ### this is kind of nasty. We need the full URI for the host/handler we are connecting to # to properly use urllib2 to make the request. This does not mesh completely cleanly # with xmlrpclib's initialization of ServerProxy. self.uri = uri self.proxyUrl = proxyUrl self.proxyUser = proxyUser self.proxyPass = proxyPass def request(self, host, handler, request_body, verbose=0): # issue XML-RPC request h = self.make_connection(host) self.set_verbosity(h, verbose) self.send_request(h, handler, request_body) self.send_host(h, host) self.send_user_agent(h) self.send_content(h, request_body) errcode, errmsg, headers = self.get_reply(h) if errcode != 200: raise ProtocolError( host + handler, errcode, errmsg, headers ) self.verbose = verbose return self.parse_response(self.get_file(h)) def make_connection(self, host, verbose=0): return _TransportConnection() def set_verbosity(self, connection, verbose): connection.verbose = verbose def send_request(self, connection, handler, request_body): connection.request = urllib2.Request(self.uri, request_body) def send_host(self, connection, host): connection.request.add_header("Host", host) def send_user_agent(self, connection): # There is no way to override the 'user-agent' sent by the UrlOpener. # This will cause a second User-agent header to be sent. # This is both different from the urllib2 documentation of add_header() # and would seem to be a bug. # # connection.request.add_header("User-agent", self.user_agent) pass def send_content(self, connection, request_body): connection.request.add_header("Content-Type", "text/xml") def get_reply(self, connection): proxyHandler = None if self.proxyUrl: if self.proxyUser: type, rest = splittype(self.proxyUrl) host, rest = splithost(rest) if self.proxyPass: user = "%s:%s" % (self.proxyUser, self.proxyPass) else: user = self.proxyUser uri = "%s://%s@%s%s" % (type, user, host, rest) else: uri = self.proxyUrl proxies = {'http':uri, 'https':uri} proxyHandler = urllib2.ProxyHandler(proxies) opener = urllib2.build_opener(proxyHandler) _fixUpHandlers(opener) try: connection.response = opener.open(connection.request) except urllib2.HTTPError, c: return c.code, c.msg, c.headers return 200, "OK", connection.response.headers def get_file(self, connection): return connection.response

1 0

New and Improved Import Hooks
by Moore, Paul Dec. 6, 2002

Dec. 6, 2002

> It doesn't matter much, though, in that as long as we > don't want to rewrite import.c as a whole we're stuck > with the two-step scheme anyway. I don't wish to seem difficult here, but didn't this start from the basis that import.c was a mess, Jim Ahlstrom's patch was going to make it worse, and we should clean it up rather than making it worse? Don't get me wrong, I'd like to see a better import hook mechanism. And I'd like to see zipfile imports. And I'd like to see the import code cleaned up. But iu.py does all I need from import hooks. Jim Ahlstrom's patch does zip imports. If Just's code doesn't clean up the import code, I quite honestly don't understand the point. I think this reinforces the need for a PEP. It would clearly describe the benefits of the patch - both in overall terms, and in comparison to the (more limited) scope of PEP 273. Paul.

2 1