txdlo
is a Python package that provides a class called DeferredListObserver
.
As you might guess, DeferredListObserver
lets you observe callback and
errback events from a list of Twisted
deferreds. You
can add observers that will be passed information about deferreds firing.
You can add deferreds to the observed list at any time, which is very
useful if you're dynamically creating deferreds that you want to monitor.
The class can be used to easily build functions or classes that provide deferreds that fire when arbitrary combinations of events from the observed deferreds have occurred.
For example you can write functions or classes that support deferreds that
- Implement Twisted's
DeferredList
or simple variants of it, or that let you separate the various behaviors ofDeferredList
into simpler functions. - Provide a deferred that fires when N of the observed deferreds have fired.
- Provide a deferred that ignores errors until one of the observed deferred succeeds, only firing with an error if all the observed deferreds fail.
- Or (a more involved example), suppose you have 3 methods that can return you a user's avatar: a fast local cache, a filesystem, and a slow network call to Gravatar. You want to write a deferred-returning function that launches all three lookups at once and fires its deferred with the first answer. But if the cache and/or filesystems fails first, you don't want to trigger an error, you instead want to take the result from Gravatar and add it to the cache and/or filesystem, as well firing the returned deferred with the result (wherever it comes from). Only if all three lookups fail do you want to errback the deferred you returned.
Here's a simplified version of Twisted's DeferredList class, written as a function.
from twisted.internet.defer import Deferred, succeed
from txdlo import DeferredListObserver
def deferredList(deferreds):
"""
Return a deferred that fires with a list of (success, result) tuples,
'success' being a boolean.
@param deferreds: a C{list} of deferreds.
@return: a L{twisted.internet.defer.Deferred} that fires as above.
"""
if len(deferreds) == 0:
return succeed([])
dlo = DeferredListObserver(maintainHistory=True)
map(dlo.append, deferreds)
deferred = Deferred()
def observer(*ignore):
if dlo.pendingCount == 0:
# Everything in the list has fired.
resultList = [None] * len(deferreds)
for index, success, value in dlo.history:
resultList[index] = (success, value)
deferred.callback(resultList)
dlo.observe(observer)
return deferred
A DeferredListObserver
maintains three counts:
successCount
: the number of observed deferreds that have fired successfully.failureCount
: the number of observed deferreds that have errored.pendingCount
: the number of observed deferreds that have not yet fired.
As in the example above, you can arrange for your observer function to
examine the DeferredListObserver
to see its state and act accordingly.
A DeferredListObserver
can maintain the history of events it has seen (as
in the example above). This can serve two purposes: (1) it can be useful
for an observer function to have easy access to the results of the
deferreds without having to store the results itself (as above), and (2) if
you add an observer after some deferreds have already fired you may want
your observer to be called with the events it missed.
Observer functions must take 3 arguments:
index
: the index of the deferred that fired. The index is the zero-based order in which deferreds were added to theDeferredListObserver
via itsappend
function.success
:True
if the deferred in question was fired via itscallback
,False
if it was fired viaerrback
.value
: the value the deferred was fired with.
To cause an added observer to immediately be called with the event history
(if any), you must instantiate the DeferredListObserver
with
maintainHistory=True
and pass replayHistory=True
to observe
. If you
add an observer and request that the event history is replayed to it but
the DeferredListObserver
is not maintaining a history, a RuntimeError
will be raised.
Observers added to a DeferredListObserver
will be called in the order
they were added.
The (untested) code in examples.py
gives some example usages.
The unit tests in txdlo/test/test_txdlo.py
may also be instructive.
To run the unit tests, either use make test
or trial txdlo
.
Note that the DeferredListObserver
adds transparent callback and errback
functions to the deferreds it is observing. The functions will see the
state of the firing deferred at that point in its callback chain. If you
add additional callbacks or errbacks to a deferred after passing it to
DeferredListObserver.append
, what goes on in those callbacks and errbacks
will not be seen. For example, consider this code:
from twisted.internet.defer import Deferred, succeed, fail
from txdlo import DeferredListObserver
def die(value):
raise Exception()
def observer(index, success, value):
if success:
print 'The deferred succeeded!'
dlo = DeferredListObserver()
dlo.observe(observer)
deferred = succeed(42)
dlo.append(deferred) # dlo adds a transparent callback & errback to deferred.
deferred.addCallback(die)
The callback added by the DeferredListObserver
(in the .append
call)
will receive the succesful (42) value and report that the deferred fired
successfully. But the later callback (die
) for deferred
will cause the
deferred to transition into a failed state.
There's nothing that can be done about this (depending on your needs, it can be an advantage). That's just the way Twisted deferreds work. If it's a problem for you, you can avoid it by not adding callbacks to deferreds after you begin observing them.
You'll see a similar warnings on the documentation for Twisted's DeferredList
If you don't understand this warning, you should probably spend more quality time alone with Twisted deferreds!