Mailman 3 May 2000 - Python-Dev

Re: No 1.6! (was Re: A REALLY COOL PYTHON FEATURE:)
by Christian Tismer May 23, 2000

May 23, 2000

Magnus Lie Hetland wrote: > > Aahz Maruch <aahz(a)netcom.com> wrote in message > news:8fh9ki$51h$1@slb3.atl.mindspring.net... > > In article <rhgmhs07ulsob3pptd6eh4f2ag4qj911bj(a)4ax.com>, > > Ben Wolfson <rumjuggler(a)cryptarchy.org> wrote: > > > > > >', '.join(['foo', 'bar', 'baz']) > > > > This only works in Python 1.6, which is only released as an alpha at > > this point. I suggest rather strongly that we avoid 1.… [View More]6-specific idioms > > until 1.6 gets released, particularly in relation to FAQ-type questions. > > This is indeed a bit strange IMO... If I were to join the elements of a > list I would rather ask the list to do it than some string... I.e. > > ['foo', 'bar', 'baz'].join(', ') > > (...although it is the string that joins the elements in the resulting > string...) I believe the notation of "everything is an object, and objects provide all their functionality" is a bit stressed in Python 1.6 . The above example touches the limits where I'd just say "OO isn't always the right thing, and always OO is the wrong thing". A clear advantage of 1.6's string methods is that much code becomes shorter and easier to read, since the nesting level of braces is reduced quite much. The notation also appears to be more in the order of which actions are actually processed. The split/join issue is really on the edge where I begin to not like it. It is clear that the join method *must* be performed as a method of the joining character, since the method expects a list as its argument. It doesn't make sense to use a list method, since lists have nothing to do with strings. Furthermore, the argument to join can be any sequence. Adding a join method to any sequence, just since we want to join some strings would be overkill. So the " ".join(seq) notation is the only possible compromise, IMHO. It is actually arguable if this is still "Pythonic". What you want is to join a list of string by some other string. This is neither a natural method of the list, nor of the joining string in the first place. If it came to the point where the string module had some extra methods which operate on two lists of string perhaps, we would have been totally lost, and enforcing some OO method to support it would be completely off the road. Already a little strange is that the most string methods return new objects all the time, since strings are immutable. join is of really extreme design, and compared with other string functions which became more readable, I think it is counter-intuitive and not the way people are thinking. The think "I want to join this list by this string". Furthermore, you still have to import string, in order to use its constants. Instead of using a module with constants and functions, we now always have to refer to instances and use their methods. It has some benefits in simple cases. But if there are a number of different objects handled by a function, I think enforcing it to be a method of one of the objects is the wrong way, OO overdone. doing-OO-only-if-it-looks-natural-ly y'rs - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com [View Less]

10 27

ANNOUNCE: Python CVS tree moved to SourceForge
by Guido van Rossum May 22, 2000

May 22, 2000

I'm happy to announce that we've moved the Python CVS tree to SourceForge. SourceForge (www.sourceforge.net) is a free service to Open Source developers run by VA Linux. The change has two advantages for us: (1) we no longer have to deal with the mirrorring of our writable CVS repository to the read-only mirror ar cvs.python.org (which will soon be decommissioned); (2) we will be able to add new developers with checkin privileges. In addition, we benefit from the high visibility and … [View More]

2 1

Re: [Python-Dev] repr vs. str and locales again
by M.-A. Lemburg May 22, 2000

May 22, 2000

Fredrik Lundh wrote: > > M.-A. Lemburg <mal(a)lemburg.com> wrote: > > > But then how can this have any effect on squishdot's indexing? > > > > The only possible reason I can see is that this squishdot > > application uses 'print' to write the data -- perhaps > > it pipes it through some other tool ? > > but doesn't the patch only affects code that manages to call tp_print > without the PRINT_RAW flag? (that is, in "repr" mode rather than "… [View More]

1 0

HTTP/1.1 capable httplib module
by Jeremy Hylton May 22, 2000

May 22, 2000

I applied the recent changes to the CVS httplib to Greg's httplib (call it httplib11) this afternoon. The result is included below. I think this is quite close to checking in, but it could use a slightly better test suite. There are a few outstanding questions. httplib11 does not implement the debuglevel feature. I don't think it's important, but it is currently documented and may be used. Guido, should we implement it? httplib w/SSL uses a constructor with this prototype: def … [View More]__init__(self, host='', port=None, **x509): It looks like the x509 dictionary should contain two variables -- key_file and cert_file. Since we know what the entries are, why not make them explicit? def __init__(self, host='', port=None, cert_file=None, key_file=None): (Or reverse the two arguments if that is clearer.) The FakeSocket class in CVS has a comment after the makefile def line that says "hopefully, never have to write." It won't do at all the right thing when called with a write mode, so it ought to raise an exception. Any reason it doesn't? I'd like to add a couple of test cases that use HTTP/1.1 to get some pages from python.org, including one that uses the chunked encoding. Just haven't gotten around to it. Question on that front: Does it make sense to incorporate the test function in the module with the std regression test suite? In general, I would think so. In this particular case, the test could fail because of host networking problems. I think that's okay as long as the error message is clear enough. Jeremy """HTTP/1.1 client library""" # Written by Greg Stein. import socket import string import mimetools try: from cStringIO import StringIO except ImportError: from StringIO import StringIO error = 'httplib.error' HTTP_PORT = 80 HTTPS_PORT = 443 class HTTPResponse(mimetools.Message): __super_init = mimetools.Message.__init__ def __init__(self, fp, version, errcode): self.__super_init(fp, 0) if version == 'HTTP/1.0': self.version = 10 elif version[:7] == 'HTTP/1.': self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 else: raise error, 'unknown HTTP protocol' # are we using the chunked-style of transfer encoding? tr_enc = self.getheader('transfer-encoding') if tr_enc: if string.lower(tr_enc) != 'chunked': raise error, 'unknown transfer-encoding' self.chunked = 1 self.chunk_left = None else: self.chunked = 0 # will the connection close at the end of the response? conn = self.getheader('connection') if conn: conn = string.lower(conn) # a "Connection: close" will always close the # connection. if we don't see that and this is not # HTTP/1.1, then the connection will close unless we see a # Keep-Alive header. self.will_close = string.find(conn, 'close') != -1 or \ ( self.version != 11 and \ not self.getheader('keep-alive') ) else: # for HTTP/1.1, the connection will always remain open # otherwise, it will remain open IFF we see a Keep-Alive header self.will_close = self.version != 11 and \ not self.getheader('keep-alive') # do we have a Content-Length? # NOTE: RFC 2616, S4.4, #3 says we ignore this if tr_enc is "chunked" length = self.getheader('content-length') if length and not self.chunked: self.length = int(length) else: self.length = None # does the body have a fixed length? (of zero) if (errcode == 204 or # No Content errcode == 304 or # Not Modified 100 <= errcode < 200): # 1xx codes self.length = 0 # if the connection remains open, and we aren't using chunked, and # a content-length was not provided, then assume that the connection # WILL close. if not self.will_close and \ not self.chunked and \ self.length is None: self.will_close = 1 # if there is no body, then close NOW. read() may never be # called, thus we will never mark self as closed. if self.length == 0: self.close() def close(self): if self.fp: self.fp.close() self.fp = None def isclosed(self): # NOTE: it is possible that we will not ever call self.close(). This # case occurs when will_close is TRUE, length is None, and we # read up to the last byte, but NOT past it. # # IMPLIES: if will_close is FALSE, then self.close() will ALWAYS be # called, meaning self.isclosed() is meaningful. return self.fp is None def read(self, amt=None): if self.fp is None: return '' if self.chunked: chunk_left = self.chunk_left value = '' while 1: if chunk_left is None: line = self.fp.readline() i = string.find(line, ';') if i >= 0: line = line[:i] # strip chunk-extensions chunk_left = string.atoi(line, 16) if chunk_left == 0: break if amt is None: value = value + self.fp.read(chunk_left) elif amt < chunk_left: value = value + self.fp.read(amt) self.chunk_left = chunk_left - amt return value elif amt == chunk_left: value = value + self.fp.read(amt) self.fp.read(2) # toss the CRLF at the end of the chunk self.chunk_left = None return value else: value = value + self.fp.read(chunk_left) amt = amt - chunk_left # we read the whole chunk, get another self.fp.read(2) # toss the CRLF at the end of the chunk chunk_left = None # read and discard trailer up to the CRLF terminator ### note: we shouldn't have any trailers! while 1: line = self.fp.readline() if line == '\r\n': break # we read everything; close the "file" self.close() return value elif amt is None: # unbounded read if self.will_close: s = self.fp.read() else: s = self.fp.read(self.length) self.close() # we read everything return s if self.length is not None: if amt > self.length: # clip the read to the "end of response" amt = self.length self.length = self.length - amt s = self.fp.read(amt) # close our "file" if we know we should ### I'm not sure about the len(s) < amt part; we should be ### safe because we shouldn't be using non-blocking sockets if self.length == 0 or len(s) < amt: self.close() return s class HTTPConnection: _http_vsn = 11 _http_vsn_str = 'HTTP/1.1' response_class = HTTPResponse default_port = HTTP_PORT def __init__(self, host, port=None): self.sock = None self.response = None self._set_hostport(host, port) def _set_hostport(self, host, port): if port is None: i = string.find(host, ':') if i >= 0: port = int(host[i+1:]) host = host[:i] else: port = self.default_port self.host = host self.port = port self.addr = host, port def connect(self): """Connect to the host and port specified in __init__.""" self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.sock.connect(self.addr) def close(self): """Close the connection to the HTTP server.""" if self.sock: self.sock.close() # close it manually... there may be other refs self.sock = None if self.response: self.response.close() self.response = None def send(self, str): """Send `str' to the server.""" if self.sock is None: self.connect() # send the data to the server. if we get a broken pipe, then close # the socket. we want to reconnect when somebody tries to send again. # # NOTE: we DO propagate the error, though, because we cannot simply # ignore the error... the caller will know if they can retry. try: self.sock.send(str) except socket.error, v: if v[0] == 32: # Broken pipe self.close() raise def putrequest(self, method, url): """Send a request to the server. `method' specifies an HTTP request method, e.g. 'GET'. `url' specifies the object being requested, e.g. '/index.html'. """ if self.response is not None: if not self.response.isclosed(): ### implies half-duplex! raise error, 'prior response has not been fully handled' self.response = None if not url: url = '/' str = '%s %s %s\r\n' % (method, url, self._http_vsn_str) try: self.send(str) except socket.error, v: if v[0] != 32: # Broken pipe raise # try one more time (the socket was closed; this will reopen) self.send(str) self.putheader('Host', self.host) if self._http_vsn == 11: # Issue some standard headers for better HTTP/1.1 compliance # note: we are assuming that clients will not attempt to set these # headers since *this* library must deal with the consequences. # this also means that when the supporting libraries are # updated to recognize other forms, then this code should be # changed (removed or updated). # we only want a Content-Encoding of "identity" since we don't # support encodings such as x-gzip or x-deflate. self.putheader('Accept-Encoding', 'identity') # we can accept "chunked" Transfer-Encodings, but no others # NOTE: no TE header implies *only* "chunked" #self.putheader('TE', 'chunked') # if TE is supplied in the header, then it must appear in a # Connection header. #self.putheader('Connection', 'TE') else: # For HTTP/1.0, the server will assume "not chunked" pass def putheader(self, header, value): """Send a request header line to the server. For example: h.putheader('Accept', 'text/html') """ str = '%s: %s\r\n' % (header, value) self.send(str) def endheaders(self): """Indicate that the last header line has been sent to the server.""" self.send('\r\n') def request(self, method, url, body=None, headers={}): """Send a complete request to the server.""" try: self._send_request(method, url, body, headers) except socket.error, v: if v[0] != 32: # Broken pipe raise # try one more time self._send_request(method, url, body, headers) def _send_request(self, method, url, body, headers): self.putrequest(method, url) if body: self.putheader('Content-Length', str(len(body))) for hdr, value in headers.items(): self.putheader(hdr, value) self.endheaders() if body: self.send(body) def getreply(self): """Get a reply from the server. Returns a tuple consisting of: - server response code (e.g. '200' if all goes well) - server response string corresponding to response code - any RFC822 headers in the response from the server """ file = self.sock.makefile('rb') line = file.readline() try: [ver, code, msg] = string.split(line, None, 2) except ValueError: try: [ver, code] = string.split(line, None, 1) msg = "" except ValueError: self.close() return -1, line, file if ver[:5] != 'HTTP/': self.close() return -1, line, file errcode = int(code) errmsg = string.strip(msg) response = self.response_class(file, ver, errcode) if response.will_close: # this effectively passes the connection to the response self.close() else: # remember this, so we can tell when it is complete self.response = response return errcode, errmsg, response class FakeSocket: def __init__(self, sock, ssl): self.__sock = sock self.__ssl = ssl return def makefile(self, mode): # hopefully, never have to write # XXX add assert about mode != w??? msgbuf = "" while 1: try: msgbuf = msgbuf + self.__ssl.read() except socket.sslerror, msg: break return StringIO(msgbuf) def send(self, stuff, flags = 0): return self.__ssl.write(stuff) def recv(self, len = 1024, flags = 0): return self.__ssl.read(len) def __getattr__(self, attr): return getattr(self.__sock, attr) class HTTPSConnection(HTTPConnection): """This class allows communication via SSL.""" __super_init = HTTPConnection.__init__ default_port = HTTPS_PORT def __init__(self, host, port=None, **x509): self.__super_init(host, port) self.key_file = x509.get('key_file') self.cert_file = x509.get('cert_file') def connect(self): """Connect to a host onf a given port Note: This method is automatically invoked by __init__, if a host is specified during instantiation. """ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect(self.addr) ssl = socket.ssl(sock, self.key_file, self.cert_file) self.sock = FakeSocket(sock, ssl) class HTTPMixin: """Mixin for compatibility with httplib.py from 1.5. requires that class that inherits defines the following attributes: super_init super_connect super_putheader super_getreply """ _http_vsn = 10 _http_vsn_str = 'HTTP/1.0' def connect(self, host=None, port=None): "Accept arguments to set the host/port, since the superclass doesn't." if host is not None: self._set_hostport(host, port) self.super_connect() def set_debuglevel(self, debuglevel): "The class no longer supports the debuglevel." pass def getfile(self): "Provide a getfile, since the superclass' use of HTTP/1.1 prevents it." return self.file def putheader(self, header, *values): "The superclass allows only one value argument." self.super_putheader(header, string.joinfields(values,'\r\n\t')) def getreply(self): "Compensate for an instance attribute shuffling." errcode, errmsg, response = self.super_getreply() if errcode == -1: self.file = response # response is the "file" when errcode==-1 self.headers = None return -1, errmsg, None self.headers = response self.file = response.fp return errcode, errmsg, response class HTTP(HTTPMixin, HTTPConnection): super_init = HTTPConnection.__init__ super_connect = HTTPConnection.connect super_putheader = HTTPConnection.putheader super_getreply = HTTPConnection.getreply _http_vsn = 10 _http_vsn_str = 'HTTP/1.0' def __init__(self, host='', port=None): "Provide a default host, since the superclass requires one." # Note that we may pass an empty string as the host; this will throw # an error when we attempt to connect. Presumably, the client code # will call connect before then, with a proper host. self.super_init(host, port) class HTTPS(HTTPMixin, HTTPSConnection): super_init = HTTPSConnection.__init__ super_connect = HTTPSConnection.connect super_putheader = HTTPSConnection.putheader super_getreply = HTTPSConnection.getreply _http_vsn = 10 _http_vsn_str = 'HTTP/1.0' def __init__(self, host='', port=None, **x509): "Provide a default host, since the superclass requires one." # Note that we may pass an empty string as the host; this will throw # an error when we attempt to connect. Presumably, the client code # will call connect before then, with a proper host. self.super_init(host, port, **x509) def test(): """Test this module. The test consists of retrieving and displaying the Python home page, along with the error code and error string returned by the www.python.org server. """ import sys import getopt opts, args = getopt.getopt(sys.argv[1:], 'd') dl = 0 for o, a in opts: if o == '-d': dl = dl + 1 host = 'www.python.org' selector = '/' if args[0:]: host = args[0] if args[1:]: selector = args[1] h = HTTP() h.set_debuglevel(dl) h.connect(host) h.putrequest('GET', selector) h.endheaders() errcode, errmsg, headers = h.getreply() print 'errcode =', errcode print 'errmsg =', errmsg print if headers: for header in headers.headers: print string.strip(header) print print h.getfile().read() if hasattr(socket, 'ssl'): host = 'www.c2.net' hs = HTTPS() hs.connect(host) hs.putrequest('GET', selector) hs.endheaders() errcode, errmsg, headers = hs.getreply() print 'errcode =', errcode print 'errmsg =', errmsg print if headers: for header in headers.headers: print string.strip(header) print print hs.getfile().read() if __name__ == '__main__': test() [View Less]

3 3

Is the tempfile module really a security risk?
by Guido van Rossum May 22, 2000

May 22, 2000

Every few months I receive patches that purport to make the tempfile module more secure. I've never felt that it is a problem. What is with these people? My feeling about these suggestions has always been that they have read about similar insecurities in C code run by the super-user, and are trying to get the teacher's attention by proposing something clever. Or is there really a problem? Is anyone in this forum aware of security issues with tempfile? Should I worry? Is the "random-… [View More]tempfile" patch that the poster below suggested worth applying? --Guido van Rossum (home page: http://www.python.org/~guido/) ------- Forwarded Message Date: Sun, 21 May 2000 19:34:43 +0200 From: =?iso-8859-1?Q?Ragnar_Kj=F8rstad?= <ragnark(a)vestdata.no> To: Guido van Rossum <guido(a)python.org> cc: patches(a)python.org Subject: Re: [Patches] Patch to make tempfile return random filenames On Sun, May 21, 2000 at 12:17:08PM -0700, Guido van Rossum wrote: > Hm, I don't like this very much. Random sequences have a small but > nonzero probability of generating the same number in rapid succession > -- probably one in a million or so. It would be very bad if one in a > million rums of a particular application crashed for this reason. > > A better way do prevent this kind of attack (if you care about it) is > to use mktemp.TemporaryFile(), which avoids this vulnerability in a > different way. > > (Also note the test for os.path.exists() so that an attacker would > have to use very precise timing to make this work.) 1. the path.exist part does not solve the problem. It causes a racing condition that is not very hard to get around, by having a program creating and deleting the file at maximum speed. It will have a 50% chance of breaking your program. 2. O_EXCL does not always work. E.g. it does not work over NFS - there are probably other broken implementations too. 3. Even if mktemp.TemporaryFile had been sufficient, providing mktemp in this dangerous way is not good. Many are likely to use it either not thinking about the problem at all, or assuming it's solved in the module. 4. The problems you describe can easily be overcome. I removed the counter and the file-exist check because I figgured they were no longer needed. I was wrong. Either a larger number should be used and/or counter and or file-exist check. Personally I would want the random part to bee large enough not have to worry about collisions either by chance, after a fork, or by deliberate attack. Do you want a new patch that adresses theese problems better? - -- Ragnar Kjørstad _______________________________________________ Patches mailing list Patches(a)python.org http://www.python.org/mailman/listinfo/patches ------- End of Forwarded Message [View Less]

2 1

Heads up: unicode file I/O in JPython.
by bckfnn＠worldonline.dk May 20, 2000

May 20, 2000

I have recently released errata-07 which improves on JPython's ability to handle unicode characters as well as binary data read from and written to python files. The conversions can be described as - I/O to a file opened in binary mode will read/write the low 8-bit of each char. Writing Unicode chars >0xFF will cause silent truncation [*]. - I/O to a file opened in text mode will push the character through the default encoding for the platform (in addition to handling CR/LF … [View More]

1 0

Dynamic linking problem on Solaris
by Greg Ward May 19, 2000

May 19, 2000

Hi all -- interesting problem with building Robin Dunn's extension for BSD DB 2.x as a shared object on Solaris 2.6 for Python 1.5.2 with GCC 2.8.1 and Sun's linker. (Yes, all of those things seem to matter.) DB 2.x (well, at least 2.7.7) contains this line of C code: *mbytesp = sb.st_size / MEGABYTE; where 'sb' is a 'struct stat' -- ie. 'sb.st_size' is a long long, which I believe is 64 bits on Solaris. Anyways, GCC compiles this division into a subroutine call -- I guess the SPARC … [View More]doesn't have a 64-bit divide, or if it does then GCC doesn't know about it. Of course, the subroutine in question -- '__cmpdi2' -- is defined in libgcc.a. So if you write a C application that uses BSD DB 2.x, and compile and link it with GCC, no problem -- everything is controlled by GCC, so libgcc.a gets linked in at the appropriate time, the linker finds '__cmpdi2' and includes it in your binary executable, and everything works. However, if you're building a Python extension that uses BSD DB 2.x, there's a problem: the default command for creating a shared extension on Solaris is "ld -G" -- this is in Python's Makefile, so it affects extension building with either Makefile.pre.in or the Distutils. However, since "ld" is Sun's "ld", it doesn't know anything about libgcc.a. And, since presumably no 64-bit division is done in Python itself, '__cmpdi2' isn't already present in the Python binary. The result: when you attempt to load the extension, you die: $ python -c "import dbc" Traceback (innermost last): File "<string>", line 1, in ? ImportError: ld.so.1: python: fatal: relocation error: file ./dbcmodule.so: symbol __cmpdi2: referenced symbol not found The workaround turns out to be fairly easy, and there are actually two of them. First, add libgcc.a to the link command, ie. instead of ld -G db_wrap.o -L/usr/local/BerkeleyDB/lib -ldb -o dbcmodule.so use ld -G db_wrap.o -L/usr/local/BerkeleyDB/lib -ldb \ /depot/gnu/plat/lib/gcc-lib/sparc-sun-solaris2.6/2.8.1/libgcc.a \ -o dbcmodule.so (where the location of libgcc.a is variable, but invariably hairy). Or, it turns out that you can just use "gcc -G" to create the extension: gcc -G db_wrap.o -ldb -o dbcmodule.so Seems to me that the latter is a no-brainer. So the question arises: why is the default command for building extensions on Solaris "ld -G" instead of "gcc -G"? I'm inclined to go edit my installed Makefile to make this permanent... what will that break? Greg -- Greg Ward - software developer gward(a)mems-exchange.org MEMS Exchange / CNRI voice: +1-703-262-5376 Reston, Virginia, USA fax: +1-703-262-5367 [View Less]

1 0

Re: [Python-Dev] hey, who broke the array module?
by Jack Jansen May 18, 2000

May 18, 2000

> I broke it with my patches to test overflow for some of the PyArg_Parse*() > formatting characters. The upshot of testing for overflow is that now those > formatting characters ('b', 'h', 'i', 'l') enforce signed-ness or > unsigned-ness as appropriate (you have to know if the value is signed or > unsigned to know what limits to check against for overflow). Two > possibilities presented themselves: I think this is a _very_ bad idea. I have a few thousand (literally) routines … [View More]

3 2

Re: [Python-Dev] hey, who broke the array module?
by Trent Mick May 18, 2000

May 18, 2000

On Thu, May 18, 2000 at 03:49:59PM -0700, Guido van Rossum wrote: > > Maybe we can come up with a modifier for signed or unsigned range > checking? Ha! How about 'u'? :) Or 's'? :) I really can't think of a nice answer for this. Could introduce completely separate formatter characters that do the range checking and remove range checking from the current formatters. That is an ugly kludge. Could introduce a separate PyArg_CheckedParse*() or something like that and slowly migrate to … [View More]

1 0

Attempt script name with '.py' appended instead of failing?
by Mark Hammond May 17, 2000

May 17, 2000

For about the 1,000,000th time in my life (no exaggeration :-), I just typed "python.exe foo" - I forgot the .py. It would seem a simple and useful change to append a ".py" extension and try-again, instead of dieing the first time around - ie, all we would be changing is that we continue to run where we previously failed. Is there a good reason why we dont do this? Mark.

3 2