暨南大學(xué)并行計(jì)算實(shí)驗(yàn)室MapReduce研究現(xiàn)狀

*,Click to edit the title text format,Click to edit the outline text format,Second Outline Level,Third Outline Level,Fourth Outline Level,Fifth Outline Level,Sixth Outline Level,Seventh Outline Level,Eighth Outline Level,Ninth Outline Level,*,Click to edit the title text format,Click to edit the outline text format,Second Outline Level,Third Outline Level,Fourth Outline Level,Fifth Outline Level,Sixth Outline Level,Seventh Outline Level,Eighth Outline Level,Ninth Outline Level,暨南大學(xué)并行計(jì)算實(shí)驗(yàn)室,MapReduce,研究現(xiàn)狀,專 業(yè)：計(jì)算機(jī)軟件與理論,姓 名：周敏 丁光華,指導(dǎo)教師：周繼鵬 教授,摘要,MapReduce,研究,調(diào)試、監(jiān)控等,優(yōu)化、擴(kuò)展等,常用,API,Hadoop,改造,數(shù)據(jù)挖掘項(xiàng)目,Redpoll,Canopy,k-means,Naive bayes,SVM,調(diào)試,標(biāo)準(zhǔn)輸出,標(biāo)準(zhǔn)出錯(cuò),Web,顯示,(50030,50060,50070),NameNode,JobTracker,DataNode,TaskTracker,日志,本地重現(xiàn),:Local Runner,DistributedCache,中放入調(diào)試代碼,Profiling,目的：查性能瓶頸，內(nèi)存泄漏，線程死鎖等,工具：,jmap,jstat,hprof,jconsole,jprofiler mat,jstack,對(duì),JobTracker,的,Profile,對(duì)各,slave,節(jié)點(diǎn),TaskTracker,的,Profile,對(duì)各,slave,節(jié)點(diǎn)某,Child,進(jìn)程的,Profile(,可能存在單點(diǎn)執(zhí)行速度過慢,),監(jiān)控,目的：監(jiān)控集群或單個(gè)節(jié)點(diǎn),I/O,內(nèi)存及,CPU,工具：,Ganglia,調(diào)優(yōu)點(diǎn),(1),I/O,Shuffle,調(diào)優(yōu)點(diǎn),(2),數(shù)據(jù)壓縮,推測(cè),性執(zhí)行,(,同時(shí)執(zhí)行同一,Task,殺死運(yùn)行慢的,),同一節(jié)點(diǎn)的,Child,重用,jvm,重寫,Partitioner,使分布到各,Reducer,的數(shù)據(jù)均勻,設(shè)置堆空間大小,常用,API,Mapper,Reducer,Writable,ComparableWritable,InputFormat,OutputFormat,Partitioner,Comparator,DistributedCache,Streaming(bash/python),Hadoop,改造,JobTracker,與作業(yè)調(diào)度耦合性太強(qiáng),JobHistory,應(yīng)獨(dú)立為一個(gè),jvm,進(jìn)程,邏輯不應(yīng)與,JobTracker,耦合太強(qiáng),在,HDFS,之上整合,MPI,統(tǒng)一作業(yè)調(diào)度,Shuffle,過程只需一次,I/O,單塊磁盤失效導(dǎo)致整個(gè)節(jié)點(diǎn)失效問題,(,改,DFSClient),Hadoop,改造,文件系統(tǒng)兼容,posix,使,Map,的,key,輸出不排序,只分區(qū),NameNode,單點(diǎn)故障問題,RPC,支持大數(shù)據(jù),(,如文件,),傳輸,集群資源分配,權(quán)限管理,大規(guī)模數(shù)據(jù)挖掘,:Redpoll,文本數(shù)據(jù)挖掘,分布式分詞,分布式向量空間模型,距離度量,語料,搜狗新聞,20 news group,wikipedia,前提：,假定一個(gè)屬性值對(duì)分類的影響?yīng)毩⒂谄渌麑傩缘闹?。（類條件獨(dú)立）,樸素貝葉斯分類工作過程,每個(gè)數(shù)據(jù)樣本用一個(gè),n,維特征向量 表示，分別描述對(duì),n,個(gè)屬性 樣本的,n,個(gè)度量,假設(shè)有,m,個(gè)類 。給定一個(gè)未知的數(shù)據(jù)樣本,X,，分類法將預(yù)測(cè)具有最高后驗(yàn)概率（條件,X,下）的類。即是找最大化的 。根據(jù)貝葉斯定理有,樸素貝葉斯分類,P(X),對(duì)所有類為常數(shù)，最大化 ，對(duì) 的考慮分析：等概率，或,類條件獨(dú)立的樸素假定：，,（,k=1,，,2,，,n,）可以由訓(xùn)練樣本估值,是分類屬性，則根據(jù)樣本估值,是連續(xù)值屬性，則通常假定其服從高斯分布，因而,樸素貝葉斯分類,(,續(xù),),Canopy,大容量,高維數(shù)據(jù)集聚類,使用兩步聚類,不同的距離度量,節(jié)省計(jì)算時(shí)間,適用范圍較廣,K-means,EM,GAC,大規(guī)模支持向量機(jī),解的稀疏性及問題的凸性,將大規(guī)模的原問題分解成小規(guī)模的子問題,迭代求解子問題,直到收斂至原問題的解,.,選塊算法,分解算法,序列最小最優(yōu)化法,(sequential minimal optimization,SMO),并行實(shí)現(xiàn),Thinking in MapReduce,B,A,D,A,A,C,B,C,B,C,D,Group,Co-group,Function,Stream Flow,Filter,Filter,Aggregate,謝謝,!,