Mailman 3 September 2014 - Twisted

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Sept. 14, 2014

Sept. 14, 2014

1 0

[Twisted-Python] workflow
by Daniel Sank Sept. 12, 2014

Sept. 12, 2014

I would greatly benefit from some direction regarding workflow. In particular, how do you guys deal with having an installed version of twisted, and a version (versions?) with which to tinker? Currently, I have twisted installed in the normal end user way, and I have a clone of my fork of the git repo. I mess around in the git repo clone, but I don't know how to e.g. run the tests or run applications using the git clone. glyph explained this to me on irc a while back. Are the instructions from that recorded somehwere? Why am I asking? Because I want to do reviews and I'm not doing it because I don't know how to set up a sane environment. -- Daniel Sank

2 1

[Twisted-Python] epoll 2.4 kernel
by Reuben Hawkins Sept. 11, 2014

Sept. 11, 2014

Not sure if this is the right mailing list for this.... epoll should *not* be used on 2.4 kernels... I have 2.4 kernel running on a 32-bit x86 machine and another 2.4 kernel running on 64-bit x64 and I'm having an issue with twisted (via buildbot). I can import epoll from select on the 64-bit machine (it *should* fail but doesn't) and this later causes twisted to fail... >>> from select import epoll >>> epoll(1) Traceback (most recent call last): File "<stdin>", line 1, in <module> IOError: [Errno 38] Function not implemented On the 32-bit machine, however, importing epoll is properly failing... >>> from select import epoll Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name epoll >>> twisted/internet/default.py is hitting this same problem. It's incorrectly picking epoll when it shouldn’t because import epoll is working when it shouldn't. I 'fixed' the problem on my machine by adding a check method in epollreactor.py which does this... def check(): try: epoll(1) except IOError: return False return True and I import and call the check method in default.py. An alternate fix can be... in twisted/internet/default.py try: if platform.isLinux(): try: from select import epoll epoll(1) from twisted.internet.epollreactor import install except (IOError, ImportError): from twisted.internet.pollreactor import install Any insight would be good. I had to compile my own Python 2.7.8 on these old computers...maybe python is incorrectly detecting epoll in the build process? Thanks in advance, Reuben

3 2

[Twisted-Python] Python3: should paths be bytes or str?
by Wolfgang Rohdewald Sept. 10, 2014

Sept. 10, 2014

The porting guide says No byte paths in sys.path. I am not sure what this means, I would assume that file paths should always be native strings. Or does it mean that sys.path must only contain ascii bytes? doc for FilePath says On both Python 2 and Python 3, paths can only be bytes. and svn commit 35410 by itamarst changed the doc for some path functions in python/filepath.py from str to bytes but not all of them: fgrep 'type path:' filepath.py @type path: L{str} @type path: L{str} @type path: L{bytes} @type path: L{bytes} @type path: L{bytes} I stumbled upon this while trying to find out how much work it might be to make bin/trial run with python3 admin/run-python3-tests already passes for all twisted.spread related tests but I still need to clean up a lot. after adding an assert to FilePath.__init__, python3 bin/trial ... gives File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in run config.parseOptions() File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in parseOptions self.postOptions() File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in postOptions _BasicOptions.postOptions(self) File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in postOptions self['reporter'] = self._loadReporterByName(self['reporter']) File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in _loadReporterByName for p in plugin.getPlugins(itrial.IReporter): File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in getPlugins allDropins = getCache(package) File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in getCache mod = getModule(module.__name__) File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in getModule return theSystemPath[moduleName] File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in __getitem__ self._findEntryPathString(moduleObject)), File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in _findEntryPathString if _isPackagePath(FilePath(topPackageObj.__file__)): File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, in __init__ assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) AssertionError: path must be bytes: '/home/wr/ssdsrc/Twisted/twisted/__init__.py' -- Wolfgang

4 6

[Twisted-Python] a brief note about buildbots
by Glyph Lefkowitz Sept. 10, 2014

Sept. 10, 2014

Hi Twistosophists, Some of you doing reviews (you ARE all doing code reviews, right!?!?) might have noticed that some of the boxes on the left side of of the supported builds page were dark grey, meaning that they were offline. I spent some time over the last couple of days doing some administrative housekeeping and rebooting computers, and all of our supported builders are now back online! This makes me feel much better about the possibility of doing a release again someday :-). Speaking of colors of boxes on that page, though, we should probably try to turn some of them from red to green... -glyph P.S.: Thanks to Rackspace for providing both (A) the cloud resources we're running many these tests on, and (B) my time to do this sort of maintenance.

3 2

[Twisted-Python] jelly and instances
by Wolfgang Rohdewald Sept. 10, 2014

Sept. 10, 2014

doc says: Instance: s is an instance of UserString.UserString, with a __dict__ {'data': 'hello'}:: [\"UserString.UserString\", ['dictionary', ['data', 'hello']]] There is no test for instances. To be precise, no test ever enters jelly._unjelly_instance() I also cannot find any place in jelly.py where an Instance would be jellied. There is instance_atom = 'instance' but instance_atom is not used anywhere. Is this outdated code? If not, could you please give me a code example using this? -- Wolfgang

2 1

[Twisted-Python] pb.Copyable knowledge barrier
by Kevin Mcintyre Sept. 8, 2014

Sept. 8, 2014

goal: Give knowitall.py awareness of wordsmith.py addition. to run ./library.sh # opens 7999 for remote_dictionary and remote_define methods then both: python knowitall.py python wordsmith.py library.sh due to main/jelly issues

4 12

[Twisted-Python] PEP3131: non-ascii identifiers
by Wolfgang Rohdewald Sept. 8, 2014

Sept. 8, 2014

This does not seem to be supported by Python yet. Should that be enabled at all? If one process with PY3 sends such identifiers to a separate process with PY2, that will fail. I am not sure if that would be a problem, whoever uses this must make sure PY3 is used everywhere. If this should be forbidden, I will add a test to test_pb for this. And of course somebody should document that somewhere. There more PEP3131 is used, the more users will fall into this trap. If this should be enabled (which I think is not difficult, at least for pb): At least the patch below will be needed (only for PY3), maybe it is already sufficient. Given that nativeString and networkString are always used (done that for pb). networkString may then return bytes with the high bit set But since networkString is called in many places I want to ask and make sure that it may really be changed this way. https://twistedmatrix.com/documents/14.0.0/core/specifications/banana.html does not speak against it, so I wonder why networkString has that limitation to 7bit. concrete banana-encoded example, from modified test_pb: (the method name is getSimpleä) test_pb still passes with patched nativeString/networkString (but I only have one test for this so far, test_refcount). b'\x07\x80\x07\x82message\x01\x81\x03\x82foo\x0b\x82getSimple\xc3\xa4\x01\x81\x01\x80\x05\x82tuple\x01\x80\n\x82dictionary' diff --git twisted/python/compat.py twisted/python/compat.py index 6f76c39..6919cf6 100644 --- twisted/python/compat.py +++ twisted/python/compat.py @@ -348,10 +348,9 @@ def nativeString(s): raise TypeError("%r is neither bytes nor unicode" % s) if _PY3: if isinstance(s, bytes): - return s.decode("ascii") + return s.decode("utf-8") else: - # Ensure we're limited to ASCII subset: - s.encode("ascii") + return s else: if isinstance(s, unicode): return s.encode("ascii") @@ -428,7 +427,7 @@ if _PY3: def networkString(s): if not isinstance(s, unicode): raise TypeError("Can only convert text to bytes on Python 3, I got %r" % (s,)) - return s.encode('ascii') + return s.encode('utf-8') def networkChar(integer): """ -- Wolfgang

2 5

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Sept. 7, 2014

Sept. 7, 2014

1 0

[Twisted-Python] spawnProcess - reapProcess not retrying on failures
by Adi Roiban Sept. 5, 2014

Sept. 5, 2014

Hi, While using spawnProcess on Linux I found out that when an invalid executable is called there is a corner case in which a zombie process is left until main process exists and can not be closed. I wrote a test for this but I was not able to reproduce this error in isolation, event if I run the test for 10000 times. reapProcess will always succeed from the first call. For the production code I can always reproduce the problem. Inspecting the execution thread I found out that all pipes are closed but spawned process is not closed yet. Due to this Process.maybeCallProcessEnded() will call self.reapProcess(). In my case, os.waitpid(pid, os.WNOHANG) return 0, and self.reapProcess() will just ignore this case. Process handlers is still registered in reapProcessHandlers but reapAllProcesses is no longer called. If a add a reactor.callLater(self.reapProcess) the next call to os.waitpid will succeed and the forked process is closed. ------------ What am I doing wrong? Why reapAllProcesses is no longer called, even if reapProcessHandlers is not empty? My process protocol code is here https://gist.github.com/adiroiban/bac493f00ce5e94738ce but from what I see, this should happen for any protocol. Many thanks, -- Adi Roiban

6 21