Archive Page 2

Asynchronous Networking in Python

10Feb10

More on Python for networking: a performance comparison of different asynchronous frameworks, including Twisted and Tornado. It focuses on raw performance (it is just an HTTP echo server implemented in each platform, and tested with httpperf), so it ignores the features provided…

However, it has brought to my attention two frameworks based on libevent: Gevent and Concurrence. Gevent provides some bindings for libevent for Python, adding the use of greenlets to the equation. Concurrence goes one step further by using “Lightweight-tasks-with-message-passing approach to concurrency“, providing “an easier programming model for writing high performance network applications than existing solutions (Multi-threading, Twisted, asyncore etc).”  Concurrence can be used with the greenlets library or with ‘Stackless Python’, a branch of the standard CPython that is supposed to give better performance on multi-threading programs.

All these frameworks looks really promising if you are looking for raw performance, but I stick with Twisted for my experiments. The huge number of protocols implemented by the standard distribution, its more-than-good documentation, and all the mailing lists and blogs, make Twisted a good platform for developing networking applications, even if there are other, faster alternatives on the market.

Python streaming

10Feb10

Continuing with my interest with Twisted (and, in particular, for doing media streaming), I have discovered this article on high performance streaming in Twisted. It is not quite a tutorial,it’s more like a  brief introduction to the interface for sending/receiving large amounts of data (think about big media files) without clogging your CPU and memory. However, I’m a bit concerned because it seems there is no support in Twisted for sendfile (although there is an old branch that implements it).

I have also found some code that implements RTP in Python. Maybe it is a bit old, and it is probably a bit slow, but it seems fairly complete. It is called xshtoom, it has BSD-like license, it includes some interesting utility code (like some NAT/STUN and SIP classes), and I think it could be easy to modify and adapt to my needs…

DNS server in Twisted

09Feb10

Yesterday I found this code snippet on a DNS server made with Twisted. It just forwards the DNS queries to Google’s servers, with some optional filtering or transformation in the middle. It has been really shocking to see how easy is to make your own DNS server in Python…

Ok, ok, maybe it is not as fast as a DNS server should be, but who cares? You can put a PowerDNS server in front in recursor mode, dealing with user requests on one side while it asks and caches results from your - maybe slow - Python DNS server on the other side. And, if it is really slow, you could hide the Python server from the internet with some iptables rules, so it can only be accesible from the PowerDNS server…

from twisted.internet.protocol import Factory, Protocol
from twisted.internet import reactor
from twisted.protocols import dns
from twisted.names import client, server
class SpelDnsResolver(client.Resolver):
  def filterAnswers(self, message):
    if message.trunc:
      return self.queryTCP(message.queries).addCallback(self.filterAnswers)
    else:
      if(len(message.answers) == 0):
        query = message.queries[0]
        # code to do a dns rewrite
        return self.queryUDP('8.8.8.8').addCallback(self.filterAnswers)
    return (message.answers, message.authority, message.additional)
verbosity = 0
resolver = SpelDnsResolver(servers=[('4.2.2.2', 53)])
f = server.DNSServerFactory(clients=[resolver], verbose=verbosity)
p = dns.DNSDatagramProtocol(f)
f.noisy = p.noisy = verbosity
reactor.listenUDP(5333, p)
reactor.listenTCP(5333, f)
reactor.run()

git tips

23Dec09

Now that I’m getting serious with git, I’ve found some useful tips for using it. I still find it a bit difficult to use, much harder than Mercurial or Bazaar, but it has a better integration with subversion, and that is what we use as the central repository in our project, so I will try to reach the master’s level… even when I’ve never been an expert with subversion.

Patience

11Dec09

From newteevee:

More than 81 percent of all online video viewers click away if they encounter a clip rebuffering, according to a new study by Tubemogul. The Emeryville-based video distribution and analytics startup took a close look at 192 million video streams over the course of 14 days to figure out how much rebuffers matter. The result: 6.81 percent of all streams rebuffer at some point, and around 2.5 percent rebuffer twice.

Tubemogul also measure how many times rebuffers occur across several popular CDNs. During its tests, Limelight performed the best, while Bitgravity’s streams had to reload the most, with Akamai being somewhere in the middle.

We always heard that we didn’t really need UDP or congestion control for modern video streaming because CDNs with conventional TCP was enough, but it is interesting to see how CDN are still suffering the same real-time problems than real-time transmissions: if you are not fast enough, clients just leave.




Search

Calendar

March 2010
M T W T F S S
« Feb    
1234567
891011121314
15161718192021
22232425262728
293031