Twisted October 2013

twisted@python.org

38 participants
45 discussions

Start a nNew thread

[Twisted-Python] Weekly Bug Summary

by trac＠twistedmatrix.com

8 years, 1 month

[Twisted-Python] FTP without the protocol

by Lloyd Carothers

Hi, all, I have many users that need to send data to me for archival. Large data sets in many files (10,000's of files 100s of GB per file set.) Up to this point the users use a ftp client wrapper that uploads data to my FTP server. This has become too problematic with firewalls etc. and I now have the time to write a new client/server of my own; A ftp server for uploads only. I'm looking into using twisted, but am very open to other suggestions/solutions. For such a use case I'm surprised not to find an example. I think I just need a push in the right direction. Is producers/consumers the right approach? Thanks in advance, Lloyd

8 years, 1 month

[Twisted-Python] Clean pb solution for two-way object sync?

by Daniel Sank

(Apologies for double send but I forgot the subject line) Dear Twisted users, I recently found myself implementing a design pattern that I think twisted.pb was specifically designed to address. I think I'm not using pb correctly so I'd like advice. This is a somewhat longish post because I need to describe the problem I'm trying to solve. I have done internet searches on Stack Overflow and this list but have not found the answer to my question. If I've missed something kindly direct me to the appropriate reference. I want to implement something functionally equivalent to a network chess game. I first consider how I would do this on a single computer with no network (maybe this is bad thinking). Each piece in the game is represented by an instance of class Agent. Each agent has a .graphics attribute which is an instance of a class from a GUI toolkit library or equivalent. Whenever an agent in the game needs to do something there will be business logic executed by the game objects proper (ie the agents) which will invoke methods on the .graphics objects to update the screen. This sort of structure seems natural as it allows easy integration of drag/drop, mouse click detection etc. It also nicely separates the real business logic from the GUI. Now I want to run over the network. The question is how should I set up references between the client and server objects? Surely the server will maintain a set of objects representing the pieces in the game. It seems reasonable that each user's program will have a corresponding set of objects (with .graphics attributes). The issue is, what do we mean by "corresponding" and how do these objects talk to one another? Following is my idea so far: Each instance of AgentClient has a .server attribute which is a remote reference to an instance of AgentServer, and each instance of AgentServer has a .clients attribute which is a list of remote references to instances of AgentClient. class AgentServer(pb.referenceable): def remote_move(self, targetSquare): """Handle move request from client""" if self.thisMoveIsLegal(targetSquare): self.position = targetSquare for client in self.clients: client.callRemote("move", targetSquare) def thisMoveIsLegal(self, targetSquare): <check that this is a legal move> class AgentClient(pb.referenceable): def requestMove(self, targetSquare): """Tell server we'd like to move""" self.server.callRemote("move", targetSquare) def remote_move(self, targetSquare): """Server told us we moved""" self.position = targetSquare self.graphics.setNewPosition(targetSquare) This isn't THAT bad. The client's requestMove is thin and unecessary (I put it there for illustration). Still I need to have two separate classes with corresponding methods to handle moving the piece. This seems like the kind of thing I could twisted.pb to solve more cleanly if I only would look in the right place. This problem gets even worse when I think about how to birth new in-game objects. It would have to look like this: class PlayerServer(pb.referenceable): def newAgent(self, asker): """Client told us it wants a new Agent""" if self.thisIsLegal(): a = AgentServer() self.agents.append(a) for client in self.clients: d = client.callRemote("newAgent", a) d.addCallback(lambda obj: a.clients.append(obj)) class PlayerClient(bp.referenceable): def requestNewAgent(self): """Tell the server we want to spawn a new Agent""" self.server.callRemote("newAgent", self) def newAgent(self, serverObj): a = AgentClient() self.agents.append(a) a.server = serverObj return a This just looks wrong. Any advice? Thank you in advance for your help. Regards, Daniel Sank

8 years, 1 month

[Twisted-Python] "mind" introduced strangely in pb howto

by Daniel Sank

Dear Twisted readers, I am trying to learn to ins and outs of twisted.spread.pb with the official howto as my main point of reference. When I get to the section "Managing Clients of Perspectives" the first sentence reads "In all the IPerspective uses we have shown so far, we ignored the mind argument and created a new Avatar for every connection. This is usually an easy design choice, and it works well for simple cases." This seems odd because nothing in the howto up to this point mentions a "mind argument". Also IPerspective and Avatar are only briefly introduced but never used in example up to this point. I'm not good enough with pb yet to want to suggest an alteration but this confused me enough that I thought I should bring it up. Yours sincerely -- Daniel Sank Department of Physics Broida Hall University of California Santa Barbara, CA 93117 (805)893-3899

8 years, 1 month

[Twisted-Python] epoll keep sharing state between process even after fork.

by Flint

Hi everybody I a came across a surprising problem when using the epoll based reactor. (ticket here https://twistedmatrix.com/trac/ticket/6796) As you can see on the ticket, the epoll object seems to share some state even after fork. Which means even after having forked the process, making some changes to the inherited epoll object in one process may impact the one existing in another process ! This problem is only related to epoll based reactor, poll and select behave correctly. I'm aware that some may say that this in not a twisted related problem (but an epoll issue), but I'm addressing my question here to try to figure out what would be the best workaround for this problem in a twisted based project ? And for information this is what I have already tryed: - I though about using poll or select reactor, but this is not possible, cause as i'm writing a library I do not decide which reactor will be installed by the person who want to use it. (Besides as epoll is now the default reactor installed, I certainly want my library to work with it !) - Using spawnProcess as mentioned in the ticket comment, is not an option too, in my case, I need to share some state between the main process and subprocess. (I have on object in main process space that I want to inherit in all subprocess) - I (desperately) tryed to re-initialize the reactor._poller object after each fork to set it with a new object, but it was just a very bad idea ! :) Thank you, in advance, for any possible clue ! (for information, my project is here https://github.com/Grindizer/scaletix)

8 years, 1 month

Re: [Twisted-Python] epoll keep sharing state between process even after fork.

by Flint

> > 2. Arrange for the epoll object (or FD) to be closed after fork, but > > before exec, so that the child process can't fiddle with it > > See also: > > http://bugs.python.org/issue8713 > > ...which suggests Python 3.4 added fork+exec support to multiprocessing. > On Unix and older Python versions, you're stuck with plain fork and all > the attendant horribleness. > > This is a multiprocessing bug IMHO. > Hi again, I've found this discussion https://lkml.org/lkml/2007/10/27/25 the scenarios discussed are a bit different but it sounds to me related to the same problem. which makes me think this is not a python bug but may be an epoll design poblem with fork !

8 years, 1 month

Re: [Twisted-Python] epoll keep sharing state between process even after fork.

by Flint

> > On 03:46 pm, grindizer(a)gmail.com wrote: > >Hi everybody > > > >I a came across a surprising problem when using the epoll based > >reactor. > >(ticket here https://twistedmatrix.com/trac/ticket/6796) > > > >As you can see on the ticket, the epoll object seems to share some > >state > >even after fork. > > > >[snip] > > > >- Using spawnProcess as mentioned in the ticket comment, is not an > >option > >too, in my case, I need to share some state between the main process > >and > >subprocess. (I have on object in main process space that I want to > >inherit > >in all subprocess) > > This doesn't really explain why you can't use `spawnProcess`. There are > other ways to share state between processes. Perhaps if you describe > the object you have someone can suggest a way to share it that will > still satisfy your requirements without requiring that you use the > `multiprocessing` module. > > Jean-Paul > Hi. For the moment I share a factory object, but it's the underlying "protocol class" that is really needed. The code takes a factory object, and then launch a set of worker/process. when a connection is made on the parent process, the socket handler will be sent to one of the process (using sendmsg), the process will then run reactor.adoptStreamConnection and this is when we need the protocol class. There is an example of what I would like to achieve here https://github.com/Grindizer/scaletix the example works with select or poll reactor. with spawnProcess I guess the child process will need to somehow import the factory class and should be given the way to instanciate it ? isn't it ? I don't know other way to do it actually ? thankx again.

8 years, 1 month

[Twisted-Python] twisted & USB

by Werner Thie

Aloha is anybody aware of code allowing to attach to USB devices with twisted? The only piece of code I found is the twistification of pywws (Weather Station) on https://gist.github.com/claws/2464017 The code works, but I'm scratching my head if there is a better solution to this problem, although USB across platforms seems to be a monster to tackle. Werner

8 years, 1 month

[Twisted-Python] Buildslave outage

by Tristan Seligmann

Hi all, Sorry for the short notice. My buildslaves (bot-idnar-debian{,64}) will be down for the next 8 hours or so, due to the host systems being moved to another datacentre. -- mithrandi, i Ainil en-Balandor, a faer Ambar

8 years, 1 month

Re: [Twisted-Python] Flow-control for Pipes / Unix Domains Sockets?

by Tom Sheffler

Tobias and Itamar - Thanks for sharing the conversation and statistics. Very interesting! -Tom

8 years, 1 month

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

Twisted October 2013