Simplifying Asynchronous Code in Twisted

Twisted’s “Deferred” system for performing asynchronous operations is powerful and flexible, but very hard to learn. I’m not sure my mental model of how Deferreds work is fully correct even after several years of using them.

Recently I discovered there’s a far simpler way to drive asynchronous operations in Twisted: use the @defer.inlineCallbacks decorator, where you wait on the asynchronous result as if it were coming from a Python generator. Why didn’t I see this option three years ago?

The old “traditional” manual callback style:

def print_file(file_):
    d = async_print_file(file_)
    d.addCallback(on_success)
    d.addErrback(on_failure)
 
def get_file():
    d = async_get_file()
    d.addCallback(print_file)
    d.addErrback(on_failure)

where you end up creating awkward little classes to maintain the state you need from step to step in the chain of operations, and awkwardly hooking up argument lists with functools.partial etc.

The new generator-coroutine style is far more concise:

@defer.inlineCallbacks
def get_and_print_file():
    file = yield async_get_file()
    result = yield print_file()

The “yield from” expression suspends execution until the async operation completes. Errors are handled by placing try/except blocks in reasonable places.

Getting the details right

Two complications generally appear with simple-looking asynchronous code: error handling and management of in-flight requests. The inlineCallbacks system handles errors cleanly simply by throwing an exception from the “yield” when the call fails. It does not, however, offer any way to monitor the status of in-flight requests (e.g. to prevent an unbounded pile-up of requests to a failed server that is timing out). That still must be done manually where necessary.

Leave a Reply

Your email address will not be published. Required fields are marked *