TwaL ‘Twisted as Library’
Twisted seems to have a sterling reputation as networking code. The biggest gripes I read about it are that it is big, inconsistently documented, and that it is a framework. The latter is the problem I wish to address. A game should have a game framework, a document editer should have an editer framework, and so forth; only servers should be designed around a networking frame. I would like to be able to treat the networking code as a library, calling into it when its functionality is needed, and ignoring it otherwise.
TwaL will have two parts, a canned script that runs ‘on the OS’, networking via LAN or internet with TwaL scripts on other machines, and an interface library that connects the user program (game, maybe) with the canned script. The script will be run by the client program via the subprocess library, and will communicate with the client via fully buffered pipes. The client program can read when its ready, and write when it has something to say, and know data will not be lost.
The TwaL script will connect to a provided internet address, announce itself and its group, and download a list of other group members online at the time. That list will be updated periodically, and can be requested by the client. TwaL will connect to one or more other nodes, on request, and relay data between that remote and the client.
Data will be transferred TwaL to TwaL in JSON format, and between TwaL and client in pickle format. TwaL and the client and the pipes are all trusted. The client can send filter functions pickled to TwaL for filtering inputs.
The library API:
- initialize( group )
- spawns the TwaL process, id’ed as group. The group would typically be defined by the protocol; machines using the ‘pitchersduel’ protocol would be members of the same group.
- register_server( servername )
- initiates a connection with named server. registers self with that server, and downloads list of group members. will rerequest periodically to maintain current list.
- register_local()
- broadcasts UDP to make local network aware of presence. also starts listening on UDP multicast port and adding local participants to group member list.
- get_neighborhood() => list
- request list of players from TwaL. each player entry will have the players handle and a status string (‘idle’, ‘commited’, ‘practice’…). this function can block waiting for list, if suitably recent list not cached.
- set_status( string )
- what do we tell other computers about our status
- connect( remote, connectionHandle )
- establish connection with remote computer, connectionHandle is string.
- disconnect( connectionHandle )
- end named connection
- send_msg( connections, msg )
- send the message out on each connection listed (tuple)
- receive_msg() => ( connection, msg ) or none
- get next inbound message
- pump()
- get all inbound messages, create events of them, and put in pygame event queue
Direct comments to dkeeney@travelbyroad.net , or post in the comments here.
5 April 2007 at 10:08 pm
This looks like a really neat idea. I wonder how much latency those extra buffers will introduce…
6 April 2007 at 6:06 am
Hi David,
Would you mind coming up with a different name for this project? It seems to have nothing to do with Twisted, per se, you’re merely using Twisted to implement a particular lobby and presence system. Also, while I wish you much success in your endeavors, I am not too happy about the use of the Twisted “brand” to advocate a style of design that the whole Twisted team would consider badly broken.
Twisted isn’t just for “servers”, or even just for heavily networked apps. (Also we’re calling it an “engine” these days, not a “framework”, since that seems to be a dirty word.)
http://jyte.com/cl/twisted-is-useful-for-asynchronous-processing-generally-not-just-networking
6 April 2007 at 8:02 am
Glyph:
Sure, I can find an alternate name. I mean no disparagement towards the Twisted project.
David
Whether its called an ‘engine’ vs a ‘framework’, the complaint is the same. The application only needs and allows one, and the programmer should not have to choose between gui, game, or network mainloop, especially when the focus of the application is not networking.
6 April 2007 at 10:14 am
David,
Thanks for the (pending) name-change. I didn’t feel it was disparaging – quite the opposite, in fact, it’s flattering to have stuff named for Twisted – but it was potentially misleading.
Twisted itself used to have explicit support for the PyGame mainloop. That was developed in an era long gone by, when Twisted had less serious policy about testing and support, and so the code eroded. As an alternative strategy to putting this stuff in another process, would you consider helping us to re-instate and maintain PyGame/reactor integration? The work involved would be minimal (in fact, copying the old pyui support might still work), it’s mainly an issue of writing documentation, publicizing the appropriate API, and setting up some automated tests to be sure other changes don’t break it.
Finally, you might want to have a look at AMP, a new feature in Twisted 2.5. It was explicitly designed for simple message-oriented protocols like this. It’s explicitly simple enough to have a “non-twisted” implementation in addition to the industrial-strength one in Twisted itself.