Mailman 3 May 2012 - Python-Dev

An infinite loop in dictobject.c
by Daniel Farina May 24, 2012

May 24, 2012

Hello all. I seem to be encountering somewhat rare an infinite loop in hash table probing while importing _socket, as triggered by init_socket.c in Python 2.6, as seen/patched shipped with Ubuntu 10.04 LTS. The problem only reproduces on 32 bit machines, on both -O2 and -O0 builds (which is how I have managed to retrieve the detailed stack traces below). To cut to the chase, the bottom of the stack trace invariably looks like this, in particular the "key" (and therefore "hash") value is always the same: #0 0x08088637 in lookdict_string (mp=0xa042714, key='SO_RCVTIMEO', hash=612808203) at ../Objects/dictobject.c:421 #1 0x080886cd in insertdict (mp=0xa042714, key='SO_RCVTIMEO', hash=612808203, value=20) at ../Objects/dictobject.c:450 #2 0x08088cac in PyDict_SetItem (op=<unknown at remote 0x37>, key= 'SO_RCVTIMEO', value=20) at ../Objects/dictobject.c:701 #3 0x0808b8d4 in PyDict_SetItemString (v= {'AF_INET6': 10, 'SocketType': <type at remote 0x8275e00>, 'getaddrinfo': <built-in function getaddrinfo>, 'TIPC_MEDIUM_IMPORTANCE': 1, 'htonl': <built-in function htonl>, 'AF_UNSPEC': 0, 'TIPC_DEST_DROPPABLE': 129, 'TIPC_ADDR_ID': 3, 'PF_PACKET': 17, 'AF_WANPIPE': 25, 'PACKET_OTHERHOST': 3, 'AF_AX25': 3, 'PACKET_BROADCAST': 1, 'PACKET_FASTROUTE': 6, 'TIPC_NODE_SCOPE': 3, 'inet_pton': <built-in function inet_pton>, 'AF_ATMPVC': 8, 'NETLINK_IP6_FW': 13, 'NETLINK_ROUTE': 0, 'TIPC_PUBLISHED': 1, 'TIPC_WITHDRAWN': 2, 'AF_ECONET': 19, 'AF_LLC': 26, '__name__': '_socket', 'AF_NETROM': 6, 'SOCK_RDM': 4, 'AF_IRDA': 23, 'htons': <built-in function htons>, 'SOCK_RAW': 3, 'inet_ntoa': <built-in function inet_ntoa>, 'AF_NETBEUI': 13, 'AF_NETLINK': 16, 'TIPC_WAIT_FOREVER': -1, 'AF_UNIX': 1, 'TIPC_SUB_PORTS': 1, 'HCI_TIME_STAMP': 3, 'gethostbyname_ex': <built-in function gethostbyname_ex>, 'SO_RCVBUF': 8, 'AF_APPLETALK': 5, 'SOCK_SEQPACKET': 5, 'AF_DECnet': 12, 'PACKET_OUTGOING': 4, 'SO_SNDLOWAT': 19, 'TIPC_SRC_DROPPABLE':...(truncated), key=0x81ac5fb "SO_RCVTIMEO", item=20) at ../Objects/dictobject.c:2301 #4 0x080f6c98 in PyModule_AddObject (m=<module at remote 0xb73cac8c>, name= 0x81ac5fb "SO_RCVTIMEO", o=20) at ../Python/modsupport.c:615 #5 0x080f6d0b in PyModule_AddIntConstant (m=<module at remote 0xb73cac8c>, name=0x81ac5fb "SO_RCVTIMEO", value=20) at ../Python/modsupport.c:627 #6 0x081321fd in init_socket () at ../Modules/socketmodule.c:4708 Here, we never escape from lookdict_string. The key is not in the dictionary, but at this stage Python is trying to figure out that is the case, and cannot seem to exit because of the lack of a dummy entry. Furthermore, every single reproduced case has a dictionary with a suspicious looking violation of an invariant that I believe is communicated by the source of dictobject.c, with emphasis on the values of ma_fill, ma_used, and ma_mask, which never deviate in any reproduced case. It seems like no hash table should ever get this full, per the comments in the source: $3 = {ob_refcnt = 1, ob_type = 0x81c3aa0, ma_fill = 128, ma_used = 128, ma_mask = 127, ma_table = 0xa06b4a8, ma_lookup = 0x8088564 <lookdict_string>, ma_smalltable = {{me_hash = 0, me_key = 0x0, me_value = 0x0}, {me_hash = 1023053529, me_key = '__name__', me_value = '_socket'}, {me_hash = 1679430097, me_key = 'gethostbyname', me_value = <built-in function gethostbyname>}, {me_hash = 0, me_key = 0x0, me_value = 0x0}, {me_hash = 779452068, me_key = 'gethostbyname_ex', me_value = <built-in function gethostbyname_ex>}, {me_hash = -322108099, me_key = '__doc__', me_value = None}, {me_hash = -1649837379, me_key = 'gethostbyaddr', me_value = <built-in function gethostbyaddr>}, { me_hash = 1811348911, me_key = '__package__', me_value = None}}} The Python program that is running afoul this bug is using gevent, but the stack traces suggest that all gevent is doing at the time this crashes is importing "socket", and this is done at the very, very beginning of program execution. Finally, what's especially strange is that I had gone a very long time running this exact version of Python, libraries, and application quite frequently: it suddenly started cropping up a little while ago (maybe a few weeks). It could have been just coincidence, but if there are code paths in init_socket.c that may somehow be sensitive to the network somehow, this could have been related. I also have a limited suspicion that particularly unlucky OOM (these systems are configured in a way where malloc and friends will return NULL, i.e. no overcommit on Linux) could be related. Any guiding words, known bugs, or suspicions? -- fdr

6 9

Re: [Python-Dev] [Python-checkins] peps: Added examples.
by Nick Coghlan May 24, 2012

May 24, 2012

On May 24, 2012 11:29 PM, "Eric V. Smith" <eric(a)trueblade.com> wrote: > > Possibly I am being too tricky here by modifying parent.__path__, and I > should just modify sys.path again, as you suggest. But I was trying to > show that modifying parent.__path__ will also work. Modifying namespace package __path__ attributes directly seems like a good way to accidentally break the auto-updating. We probably don't want to encourage that. Cheers, Nick. -- Sent from my phone, thus the relative brevity :)

1 0

PEP 420 - dynamic path computation is missing rationale
by Guido van Rossum May 24, 2012

May 24, 2012

I have just reviewed PEP 420 (namespace packages) and sent Eric my detailed feedback; most of it is minor or requesting for examples and I'm sure he'll fix it to my satisfaction. Generally speaking the PEP is a beacon if clarity. But I stumbled about one feature that bothers me in its specification and through its lack of rationale. This is the section on Dynamic Path Computation: (http://www.python.org/dev/peps/pep-0420/#dynamic-path-computation). The specification bothers me because it requires in-place modification of sys.path. Does this mean sys.path is no longer a plain list? I'm sure it's going to break things left and right (or at least things will be violating this requirement left and right); there has never been a similar requirement (unlike, e.g., sys.modules, which is relatively well-known for being cached in a C-level global variable). Worse, this apparently affects __path__ variables of namespace packages as well, which are now specified as an unspecified read-only iterable. (I can only guess that there is a connection between these two features -- the PEP doesn't mention one.) Again, I would be much happier with just a list. While I can imagine there being a use case for recomputing the various paths, I am much less sure that it is worth attempting to specify that this will happen *automatically* when sys.path is modified in a certain way. I'd be much happier if these constraints were struck and the recomputation had to be requested explicitly by calling some new function in sys. >From my POV, this is the only show-stopper for acceptance of PEP 420. (That is, either a rock-solid rationale should be supplied, or the constraints should be removed.) -- --Guido van Rossum (python.org/~guido)

7 35

possible bug in distutils (Mingw32CCompiler)?
by Sturla Molden May 24, 2012

May 24, 2012

Mingw32CCompiler in cygwincompiler.py emits the symbol -mno-cygwin. This is used to make Cygwin's gcc behave as mingw. As of gcc 4.6 it is not recognized by the mingw gcc compiler itself, and causes as crash. It should be removed because it is never needed for mingw (in any version), only for cross-compilation to mingw from other gcc versions. Instead, those who use CygwinCCompiler or Linux GCC to "cross-compile" to plain Win32 can set -mno-cygwin manually. It also means -mcygwin should be removed from the output of CygwinCCompiler. I think... Sturla

3 2

Re: [Python-Dev] [Python-checkins] peps: Added examples.
by Nick Coghlan May 24, 2012

May 24, 2012

On Thu, May 24, 2012 at 8:10 PM, eric.smith <python-checkins(a)python.org> wrote: > + Lib/test/namspace_pkgs Typo: s/namspace/namespace/ > +Here we add the parent directories to ``sys.path``, and show that the > +portions are correctly found:: > + > + >>> import sys > + >>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] The trailing "/parent" shouldn't be there on either of these paths. The comments that refer back to these also need the same adjustment. > + Lib/test/namspace_pkgs Same typo as above. > + # add the first two parent paths to sys.path > + >>> import sys > + >>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] Again, need to lose the last directory from these paths and the comments that refer to them. > + # now add parent3 to the parent's __path__: > + >>> parent.__path__.append('Lib/test/namespace_pkgs/parent3/parent') This modification is incorrect, it should be: sys.path.append('Lib/test/namespace_pkgs/parent3') and both parent.__path__ and parent.child.__path__ should pick up their extra portions on the next import attempt. Also, I suggest renaming "parent1", "parent2" and "parent3" to "project1", "project2" and "project3". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Python db2 installation error
by PremAnand Lakshmanan May 24, 2012

May 24, 2012

I want to install python db2 package for Python but Im unable to install it. I have installed the easy_install and Im able to successfully import the easy_install. My easy_install location :c:/python27/lib/site-packages/ My db2 egg location c:/python27/ibm_db-1.0.5-py2.7-win32.egg How would my installation command look like in the shell, I tried this command and it gives me invalid error, C:\Python27\Scripts>easy_install c:/python27/lib/site-packages/ibm_db-1.0.5-py2. 7-win32.egg error: Not a URL, existing file, or requirement spec: 'c:/python27/lib/site-pack ages/ibm_db-1.0.5-py2.7-win32.egg' -- Prem 408-393-2545

3 2

Re: [Python-Dev] [Python-checkins] cpython: Issue #14814: addition of the ipaddress module (stage 1 - code and tests)
by Terry Reedy May 23, 2012

May 23, 2012

On 5/20/2012 7:02 AM, nick.coghlan wrote: > +def ip_address(address, version=None): > + """Take an IP string/int and return an object of the correct type. > + > + Args: > + address: A string or integer, the IP address. Either IPv4 or > + IPv6 addresses may be supplied; integers less than 2**32 will > + be considered to be IPv4 by default. > + version: An Integer, 4 or 6. If set, don't try to automatically integer, not Integer > + determine what the IP address type is. important for things > + like ip_address(1), which could be IPv4, '192.0.2.1', or IPv6, > + '2001:db8::1'. I read this as saying that a version other than 4 or 6 should be an error, and not ignored as if not set. If version is set incorrectly, it is still set. I certainly would expect an error to be an error. > + > + Returns: > + An IPv4Address or IPv6Address object. > + > + Raises: > + ValueError: if the string passed isn't either a v4 or a v6 > + address. Should say "if the *address*...", and I suggest adding "or if the version is not None, 4, or 6." > + > + """ > + if version: if version is not None: ?? Do you really want to silently ignore *every* null value, like '' or []? > + if version == 4: > + return IPv4Address(address) > + elif version == 6: > + return IPv6Address(address) else: raise ValueError() ?? ... > +def ip_network(address, version=None, strict=True): > + """Take an IP string/int and return an object of the correct type. > + > + Args: > + address: A string or integer, the IP network. Either IPv4 or > + IPv6 networks may be supplied; integers less than 2**32 will > + be considered to be IPv4 by default. > + version: An Integer, if set, don't try to automatically > + determine what the IP address type is. important for things > + like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, > + '2001:db8::1/128'. Same comments > +def ip_interface(address, version=None): > + """Take an IP string/int and return an object of the correct type. > + > + Args: > + address: A string or integer, the IP address. Either IPv4 or > + IPv6 addresses may be supplied; integers less than 2**32 will > + be considered to be IPv4 by default. > + version: An Integer, if set, don't try to automatically > + determine what the IP address type is. important for things > + like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, > + '2001:db8::1/128'. ditto > + Returns: > + An IPv4Network or IPv6Network object. Interface, not Network > +def v4_int_to_packed(address): > + """The binary representation of this address. Since integers are typically implemented as strings of binary bits, a 'binary representation' could mean a string of 0s and 1s. > + > + Args: > + address: An integer representation of an IPv4 IP address. > + > + Returns: > + The binary representation of this address. The integer address packed as 4 bytes in network (big-endian) order. > + Raises: > + ValueError: If the integer is too large to be an IPv4 IP > + address. And if the address is too small (negative)? "If the integer is negative or ..." ? > + """ > + if address> _BaseV4._ALL_ONES: or address < 0? > + raise ValueError('Address too large for IPv4') > + return struct.pack('!I', address) It is true that struct will raise struct.error: argument out of range for negative addresses, but it will also also do the same for too large addresses. So either let it propagate or catch it in both cases. For the latter (assuming the max is the max 4 byte int): try: return struct.pack('!I', address) except struct.error: raise ValueError("Address negative or too large for IPv4") > + > +def v6_int_to_packed(address): > + """The binary representation of this address. Similar comments, except packed into 16 bytes > + Args: > + address: An integer representation of an IPv4 IP address. > + > + Returns: > + The binary representation of this address. > + """ > + return struct.pack('!QQ', address>> 64, address& (2**64 - 1)) Why no range check? Here you are letting struct.error propagate. > + > +def _find_address_range(addresses): > + """Find a sequence of addresses. An 'address' can in various places be a string, int, bytes, IPv4Address, or IPv6Address. For neophyte users, I think you should be clear each time you use 'address'. From the code, I conclude it here means the latter two. > + > + Args: > + addresses: a list of IPv4 or IPv6 addresses. a list of IPv#Address objects. > +def _get_prefix_length(number1, number2, bits): > + """Get the number of leading bits that are same for two numbers. > + > + Args: > + number1: an integer. > + number2: another integer. > + bits: the maximum number of bits to compare. > + > + Returns: > + The number of leading bits that are the same for two numbers. > + > + """ > + for i in range(bits): > + if number1>> i == number2>> i: This non-PEP8 spacing is awful to read. The double space after the tighter binding operator is actively deceptive. Please use if number1 >> i == number2 >> i: > + if (number>> i) % 2: ditto > +def summarize_address_range(first, last): > + Args: > + first: the first IPv4Address or IPv6Address in the range. > + last: the last IPv4Address or IPv6Address in the range. > + > + Returns: > + An iterator of the summarized IPv(4|6) network objects. Very clear as to types. > + while first_int<= last_int: PEP8: while first_int <= last_int: is *really* much easier to read. > + while nbits>= 0: ditto, etcetera through rest of file. > + while mask: > + if ip_int& 1 == 1: Instead of no space and then 2 spaces, use uniform 1 space around & if ip_int & 1 == 1: > + ip_int>>= 1 To me, augmented assignments need a space before and after even more than plain assignments, especially with underscored names. ip_int >>= 1 I am guessing that Peter dislikes putting ' ' before '<' and '>' and perhaps '&', but it makes code harder to read. Putting an extra space after is even worse. This is as far as I read. Some of the style changes could be done with global search and selective replace. --- Terry Jan Reedy

3 4

Benchmark performance...
by stefan brunthaler May 23, 2012

May 23, 2012

Hi, as Antoine pointed out in the corresponding issue (http://bugs.python.org/issue14757#msg160870), measuring/assessing real-world performance of my patch would be interesting. I mentioned that I am not aware of any relevant Python 3 program/application to report numbers for (but guess that the speedups should persist.) Since nobody came up with an answer yet, I figured it would be a good idea to ask everybody on python-dev for suggestions... Regards, --stefan

1 0

Re: [Python-Dev] [Python-checkins] peps: Added dynamic path computation rationale, specification, and discussion.
by Nick Coghlan May 23, 2012

May 23, 2012

On Wed, May 23, 2012 at 10:35 AM, eric.smith <python-checkins(a)python.org> wrote: > + 4. An attempt is made to import an ``encodings`` portion that is > + found on a path added in step 3. I'd phrase this as something like "import an encoding from an ``encodings`` portion". You don't really import namespace package portions directly - you import the modules and packages they contain. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Python-Dev] [Python-checkins] cpython (2.7): #14804: Remove [] around optional arguments with default values
by Nick Coghlan May 22, 2012

May 22, 2012

On Tue, May 22, 2012 at 6:34 PM, hynek.schlawack <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/a36666c52115 > changeset: 77102:a36666c52115 > branch: 2.7 > parent: 77099:c13066f752a8 > user: Hynek Schlawack <hs(a)ox.cx> > date: Tue May 22 10:27:40 2012 +0200 > summary: > #14804: Remove [] around optional arguments with default values > > Mostly just mechanical removal of []. In some rare cases I've pulled the > default value up into the argument list. Be a little careful with this - "[]" is the right notation when the function doesn't support keyword arguments. At least one of the updated signatures is incorrect: > diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst > --- a/Doc/library/itertools.rst > +++ b/Doc/library/itertools.rst > @@ -627,7 +627,7 @@ > break > > > -.. function:: tee(iterable[, n=2]) > +.. function:: tee(iterable, n=2) >>> itertools.tee([], n=2) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: tee() takes no keyword arguments Since calling "tee(itr, n=2)" doesn't add really any clarity over "tee(itr, 2)", it's unlikely this function will ever gain keyword argument support (since supporting keyword arguments *is* slower than supporting only positional arguments for functions written in C. The change is probably valid for the pure Python modules, and the builtins looked right, but be wary of any extension modules in the list. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1