當Deferred遇上Thread

當Deferred遇上Thread
Deferred不會自動實現將阻塞過程轉為非阻塞過程,雖然它已經有那樣的機制但還是要你去多走一步。要將阻塞過程轉為真正的非阻塞過程,那 只有借用線程。但至於線程調用你不用太擔心,twisted已為你準備好一個方便的用法。就是將Deferred綁在Thread上,這樣就有了deferToThread,味道不比牛奶加巧克力差。
deferToThread在twisted文檔的說明,洋墨水喝得比較少,就不照字面翻譯了。大概意思是將函數f放在線程裡並作為Deferred返回,args和kwargs是函數f的參數。也就是使用deferToThread返回的是一個帶線程的Deferred,並自動使用callback調用指定的函數f。相當於在線程中運行下面的代碼
d=defer.Deferred().addCallback(f)
d.callback(result)

def deferToThread(f, *args, **kwargs): (source)
Run a function in a thread and return the result as a Deferred. Parameters f The function to call.

*args positional arguments to pass to f.

**kwargs keyword arguments to pass to f.
Returns A Deferred which fires a callback with the result of f, or an errback with a twisted.python.failure.Failure if f throws an exception.

#!/usr/bin/env python
#coding=utf-8

from twisted.internet import protocol,reactor,defer
from twisted.internet.threads import deferToThread
from twisted.protocols import basic
import time,sys
syscode='big5'

class muProtocol(basic.LineReceiver):

	def connectionMade(self):
		print 'connectionMade'
		self.factory.clients[self]={}
		self.factory.clients[self]['isblankly']=True

def lineReceived(self,data):
	self.factory.clients[self]['isblankly']=False
	self.transport.write(self.factory.getSomething(data))
	if data=='q':
		self.transport.loseConnection()
		self.factory.kungfuTea(data)

class muFactory(protocol.ServerFactory):
	protocol=muProtocol

	def __init__(self):
		reactor.callLater(1,self.timeout)
		self.clients={}

	def timeout(self):
		sys.stdout.write('.')
		sys.stdout.flush()
		reactor.callLater(1,self.timeout)

	def getSomething(self,data):
		return '\n\rServer say: %s\n\r' % data

	def goodTaste(self,data):
		print u'%s:耍下太極,仲加埋嘆下功夫茶,都有排等羅。'.encode(
		syscode,'replace') % data
		time.sleep(10)
		print u'%s,打完收功。'.encode(syscode,'replace') % data
		return data

	def wrongTaste(self,data):
		print u'%s:好味道盡在kung fu tea。'.encode(syscode,'replace') % data

	def kungfuTea(self,data):
		return deferToThread(self.goodTaste,data
		).addBoth(self.wrongTaste)

reactor.listenTCP(20190,muFactory())
reactor.run()

分析下上面的代碼。
當接收到數據後調用self.factory.kungfuTea(data)
def lineReceived(self,data):
self.factory.kungfuTea(data)
工廠中的kungfuTea只是使用deferToThread將真正處理數據的過程轉goodTaste為一個線程的Deferred,剩下的東西就和Deferred操作無異。

牛奶加巧克力不錯,老板再來一杯!

原文 http://sites.google.com/site/mypynotes/twisted-1/defertothread

Deferred筆記一

Deferred的簡單套路

從twisted的文檔入手,將Deferred獨立出來簡單的說明作用與使用方法,這樣更能快速對Deferred有深刻的了解。

一、學習資料
1、主要參考資料
官方文檔 http://twistedmatrix.com/projects/core/documentation/howto/deferredindepth.html
關聯譯文 http://wiki.woodpecker.org.cn/moin/twistedTUT00?action=AttachFile&do=get&target=deferred_tutorial.html(此鏈接將會下載離線文檔)

2、關聯資源
http://wiki.woodpecker.org.cn/moin/TwistedTUT

二、Deferred的作用
Deferred的主要作用可以用一句話比較形象說明,“不要打電話給我,我會打給你的(Don’t Call Us, We’ll Call You)”。相當於做好計劃後,執行交給Deferred,完成後再找我,我可要到一邊Happy去了。
它會根據callback運行結果(正常、發生例外)從callback鏈表中選擇執行已註冊的相應的callback函數,從而實現事件方式來驅動程序。
callback的運行不影響主程序的運行,Deferred.callback執行後馬上返回。(這點有待在後面證明)

三、Deferred實例
from twisted.internet import defer
d=defer.Deferred()
實例化後就可以對Deferred進行操作。

四、callback鏈表
鏈表是用來放置callback函數,鏈表中的函數可以響應三類事件,正常結束、例外結束、任意(包括前面兩者),它們分別用addCallback、addErrback、addBoth註冊到鏈表中。也就是說函數正常結束返回調用addCallback註冊的函數,發生例外則調用addErrback註冊的函數,使用addBoth註冊的函數無論返回什 結果都會被調用。

五、callback函數註冊
除了上面提過的addCallback、addErrback、addBoth三個註冊函數外,還有另外一個addCallbacks可以同時註冊處理正常結束和例外結束的函數。

1
2
3
4
5
6
def deferredExample():
d = defer.Deferred()
d.addCallback(failAtHandlingResult)
d.addCallbacks(yesDecision, noDecisionPassthru)
d.addCallback(handleResult) # – A –
d.addErrback(handleFailure)

上面代碼中第4行的addCallbacks效果如同
d.addCallback(yesDecision)
d.addErrback(noDecisionPassthru)

另外,我們也可以將註冊過程寫成下面形式。
d=refer.Deferred().addCallback(failAtHandlingResult).addCallbacks(yesDecision, noDecisionPassthru).addCallback(handleResult).addErrback(handleFailure)

六、callback鏈執行順序
Deferred的鏈表是按順序從第一個註冊函數開始直到最後一個函數。如果發生例外時,有多個例外處理函數,則會調用發生例外函數下面第一個例外處理函數。後面的例外處理函數將不會被調用,除非第一個例外處理函數將這個例外向下傳遞。

七、其它一些特性
Deferred實例中還有另外一些函數,如pause,unpause,setTimeout,timeoutCall,chainDeferred,這些將留作下一步的測試。
另外還沒涉及到的是Deferred上一級的defer庫中的函數及其它類的使用,從一些代碼看來這些函數將會進一步有利於Deferred的應用,但這也有待後面的進上步學習。

八、總結
經過Deferred are beautiful一節學習,了解Deferred的使用方法及其事件響應機制。

創建於2009年8月31日

原文
http://sites.google.com/site/mypynotes/twisted-1/deferred
可以被gw了

Deferred Object In Twisted

Twisted uses the Deferred object to manage the callback sequence. The client application attaches a series of functions to the deferred to be called in order when the results of the asychronous request are available(this series of functions is known as a series of callbacks, or a callback chain), together with a series of functions to be called if there is an error in the asychronous request(known as a series of errbacks or an errback chain). The asychronous library code calls the first callback when the result is available, or the first errback when an error occurs, and the Deferred object then hands the results of each callback or errback function to the next function in the chain.

The Problem that Deferreds Solve

It is the second class of concurrency problem – non-computationally intensive tasks that involve an appreciable delay – that Deferreds are designed to help solve. Functions that wait on hard drive access, database access, and network access all fall into this class, although the time delay varies.

Deferreds are designed to enable Twisted programs to wait for data without hanging until that data arrives. They do this by giving a simple management interface for callbacks to libraries and applications. Libraries know that they always make their results available by calling Deferred.callback and error by calling Deferred.errback. Applications set up result handlers by attaching callbacks and errbacks to deferreds in the order they want them called.

The basic idea behind Deferreds, and other solutions to this problem, is to keep the CPU as active as possible. If one task is waiting on data, rather than have the CPU(and the program!) idle waiting for that data(a process normally called “blocking”), the program performs other operations in the meantime, and waits for some singnal that data is ready to be processed before returning to that process.

In Twisted, a function signals to the calling function that it is waiting by returning a Deferred. When the data is available, the program activeates the callbacks on that Deferred to process the data.

Deferreds – a signal that data is yet to come

In our email sending example above, a parent functions calls a function to connect to the remote server. Asynchrony requires that this connection function return without waiting for the result so that the parent function can do other things. So how does the parent function or its controlling program know that the connection doesn’t exist yet, and how does it use the connection once it does exist?

Twisted has an object that signals this situation. When the connection function returns, it signals that the operation is imcomplete by returning a twisted.internet.defer.Deferred object.

The Deferred has two purposes. The first is that it says “I am a signal that the result of whatever you wanted me to do is still pending.”. The second is that you can ask the Deferred to run things when the data does arrive.

Callbacks

The way you tell a Deferred what to do with the data once it arrives is by adding a callback – asking the Deferred to call a function once the data arrives.

One Twisted library function that returns a Deferred is twisted.web.client.getPage. In this example, we call getPage, which returns a Deferred, and we attach a callback to handle the contents of the page once the data is available:

Python代码

from twisted.web.client import getPage   
from twisted.internet import reactor   
  
def printContents(contents):   
    '''''  
    This is the 'callback' function, added to the Deferred and called by it when the promised data is available  
    '''  
  
    print "The Deferred has called printContents with the following contents:"  
    print contents    
  
    #Stop the Twisted event handling system -- this is usually handled   
    #in higher lever ways   
  
    reactor.stop()    
  
   #call getPage, which returns immediately with a Deferred, promissing to   
   #pass the page contents onto our callbacks when the contents are available   
  
deferred = getPage('http://www.google.com')   
  
#added a callback to the deferred -- request that it run printContents when   
#the page content has been downloaded   
  
deferred.addCallback(printContents)   
  
#Begin the Twisted event handling system to manage the process -- again   
# this isn't the usual way to do this   
  
reactor.run()  

from twisted.web.client import getPage
from twisted.internet import reactor

def printContents(contents):
    '''
    This is the 'callback' function, added to the Deferred and called by it when the promised data is available
    '''

    print "The Deferred has called printContents with the following contents:"
    print contents 

    #Stop the Twisted event handling system -- this is usually handled
    #in higher lever ways

    reactor.stop() 

   #call getPage, which returns immediately with a Deferred, promissing to
   #pass the page contents onto our callbacks when the contents are available

deferred = getPage('http://www.google.com')

#added a callback to the deferred -- request that it run printContents when
#the page content has been downloaded

deferred.addCallback(printContents)

#Begin the Twisted event handling system to manage the process -- again
# this isn't the usual way to do this

reactor.run()

A very common use of Deferreds is to attach two callbacks. The result of the first callback is passed to the second callback:

Python代码

from twisted.web.client import getPage   
from twisted.internet import reactor   
  
def lowerCaseContents(contents):   
     '''''  
     This is a 'callback' function, added to the Deferred and called by  
     it when the promised data is available. It converts all the data to lower  
     case.  
     '''  
  
     return contents.lower()   
  
def printContents(contents):   
  
    '''''  
    This a 'callback' function, added to the Deferred after lowerCaseContents  
    and called by it with the results of lowerCaseContents  
    '''  
    print contents   
    reactor.stop()   
  
deferred = getPage('http://www.google.com')   
  
#add two callbacks to the deferred -- request that it run lowerCaseContents   
#when the page content has been downloaded, and then run printContents with   
#the result of lowerCaseContents   
  
deferred.addCallback(lowerCaseContents)   
  
deferred.addCallback(printContents)   
  
reactor.run() 
from twisted.web.client import getPage
from twisted.internet import reactor

def lowerCaseContents(contents):
     '''
     This is a 'callback' function, added to the Deferred and called by
     it when the promised data is available. It converts all the data to lower
     case.
     '''

     return contents.lower()

def printContents(contents):

    '''
    This a 'callback' function, added to the Deferred after lowerCaseContents
    and called by it with the results of lowerCaseContents
    '''
    print contents
    reactor.stop()

deferred = getPage('http://www.google.com')

#add two callbacks to the deferred -- request that it run lowerCaseContents
#when the page content has been downloaded, and then run printContents with
#the result of lowerCaseContents

deferred.addCallback(lowerCaseContents)

deferred.addCallback(printContents)

reactor.run()

Error handling: errbacks

Just as a asynchronous function returns before its result is available, it may also return before it is possible to detect errors: failed connections, erroneous data, protocol errors, and so on. Just as you can add callbacks to a Deferred which it calls when the data you are expecting is available, you can add error handlers(errbacks) to a Deferred for it to call when an error occurs and it cannot obtain the data:

Python代码

from twisted.web.client import getPage   
from twisted.internet import reactor   
  
def errorHandler(err):   
    '''''  
    This is an 'errback' function, added to the Deferred which will call it in   
    the event of an error  
     '''  
    #this isn't a very effective handling of the error, we just print it out:   
    
    print "An error has occurred: <%s>" % str(err)   
    #and then we stop the entire process:   
    reactor.stop()   
  
def printContents(contents):   
    '''''  
    This is a 'callback' function, added to the Deferred and called by it which   
    the page content  
    '''  
    print contents   
    reactor.stop()   
  
#we request a page which doesn't exist in order to demonstrate the   
#error chain   
  
deferred = getPage("http://www.google.com/does-not-exits")   
  
#add the callback to the Deferred to handle the page content   
  
deferred.addCallback(printContents)   
  
#add the errback to the Deferred to handle any errors   
  
deferred.addErrback(errorHandler)   
  
reactor.run()  

by blog http://huangro.javaeye.com/blog/365988