最新国家开放大学电大《大数据技术导论》实验报告实验4 大数据去重.docx

最新国家开放大学电大大数据技术导论实验报告实验4大数据去重1实验目的通过HadOoP数据去重实验，学生可以掌握准备数据、伪分布式文件系统配置方法，以及在集成开发环境ECIiPSe中实现HadooP数据去重方法。2 .实验要求了解基于HadooP处理平台的大数据去重过程，理解其主要功能，并能够在HadooP环境下独立完成。(1)制订实验计划。(2)准备数据。(3)伪分布式文件系统配置。(4)在集成开发环境ECIiPSe中实现HadoOP数据去重。3 .实验内容(1)制订实验计划。(2)进入a/usr1oca1hadoopff目录。(3)准备数据。(4)修改/usr1oca1hadoopetchadoopw目录下的HadooP配置文件。(5) NanICNOd。格式化。(6)在集成开发环境Ec1ipse中实现Hadoop数据去重。4 .实验总结通过本实验，使学生了解HadoOP数据去重的特点和过程、理解MaPRCdUCe程序的执行过程，掌握Nan1CNodC的格式化方法、Hadoop的配置文件的修改和EC1iPSC开发环境下实现HadooP数据去重的方法。5 .思考拓展(1)为什么需要NameNOdC格式化？说明NamCNOdC格式化方法。(2)为什么需要数据去重？说明HadooP数据去重的主要优势。(3)结合MaPRCdUCe程序执行过程，说明HadooP数据去重是离线处理还是在线处理。(4)说明在集成开发环境Ec1ipse中实现Hadoop数据去重的主要过程。答：数据去重方法如下【Hadoop学习项目】数据去重0.项目结构.ideaBisrcmaintojavaE1hadoop_testddata_dup1icate_demo_02GDupDriver0DupMapperQDupReducer%resources曙core-sitejcm1hdfs-site.xm1hive-site.xm1g4j.propertiesCSDN星象。数据处理过程图HDFS192168.234.21192168234.23192.168.234.21192.168.234.21192168.234.21192168.234.25192168.234.21192168.234.21192168.234.26192168.234.21192168234.27192168.234.21192168.234.27192168234.21192168.234.29sp1it，9216823421、192168.234.23192168.234.211192.168.234.21192.168.234.21192168234.25J92.16S.234.21map19216823421：NU11192.168.234.23:NU11192.168234.21：NU11192.168234.21:NU11192.168.234.21:NU11192.16823425:NU11192.168.234.21:NU1119216823421NU11NU11.NU11,NU11,NU11192.168.234.23NU11192.168.234.25NU11reduce192.168234.21:NU11192168234.23:NU11192.168.234.25:NU1192168.234.2?192168.234.26192168.234.21192.168.234.2719216823421192168.234.27192168.234.219216823429jmap192168234.21:NU11A192.168234.26:NU11192168234.21:NU11192.168.234.27:NU1119216823421:NU11192.168.234.27:NU11192.168234.21:NU11(192168234.29:NU11)192.168.23421(NU11.NU11.NU11NU1119216823426(NU11192.168.23427NU11.NU11192168.234.29NU11reduce192.168234.21:NU11192.168234.26:NU11192.168234.27:NU11192.168234.29:NU11CSDN星球1.DupDriverpackagehadoop_test.data_dup1icate_demo_02;importorg.apache.hadp.conf.Configuration;importorg.apache.hadp.fs.Path;importorg.apache.hadp.io.Nu11Writab1e;importorg.apache.hadp.io.Text;importorg.apache.hadp.mapreduce.Job;importorg.apache.hadp.mapreduce.1ib.input.Fi1eInputFormat;importorg.apache.hadp.mapreduce.1ib.output.Fi1eOutputFormat;pub1icc1assDupDriver/，处理数据:192.168.234.21192.168.234.22192.168.234.21192.168.234.21192.168.234.23192.168.234.21192.168.234.21192.168.234.21192.168.234.25192.168.234.21192.168.234.21192.168.234.26192.168.234.21192.168.234.27192.168.234.21192.168.234.27192.168.234.21192.168.234.29192.168.234.21123456789101112131415161718192021222324252627282930313233192.168.234.2634192.168.234.2135192.168.234.2536192.168.234.2537192.168.234.2138192.168.234.2239192.168.234.2140*/41pub1icstaticvoidmain(Stringargs)throwsException(4243System.SetProperty("HADP-USER-NAMEm,"root");4445 Configurationconf=newConfiguration();46 Dobjob=Job.get1nstance(conf);47 job.setJarByC1ass(DupDriver.c1ass);48 job.SetMapperc1ass(DupMapper.C1ass);49 job.setReducerC1ass(DupReducer.c1ass);50 job.setMapOutputKeyC1ass(Text.c1ass);51 job.SetMapoutputva1uec1ass(Nu11Writab1e.C1ass);/VaIUe为NU11,具体原因后续会有说明52 job.setOutputKeyC1ass(Text.c1ass);53 job.setOutputVa1ueC1ass(Nu11Writab1e.c1ass);54 Fi1eInputFormat.setInputPaths(job,newPath('7hadoop-testdupdup.txt");55 Fi1eOutputFormat.setOutputPath(jobjnewPath("hadooptestdupwordcountresu1t,');56 job.waitForComp1etion(true);57 58 2. DupMapper1234567891011121314151617packagehadoop-test.data_dup1icate_denxj_02;importorg.apache.hadp.io.1ongWritab1e;importorg.apache.hadp.io.Nu11Writab1e;importorg.apache.hadp.io.Text;importorg.apache.hadp.mapreduce.Mapper;importjava.io.IOException;pub1icc1assDupMapperextendsMapper<1ongWritab1e,Text,Text,Nu11Writab1e>gOverrideprotectedvoidmap(1ongWritab1ekey.Textva1ue.Contextcontext)throwsIOException,InterruptedException/Va1Ue:192.168.70.49/因为目标为去重,不用线计数量.因此不用让key期,可减少IO时间开销context.write(newText(va1ue),Nu11Writab1e.get();183. DupReducer123456789101112131415packagehadoop-test.data_dup1icate_demo_02;importorg.apache.hadp.io.Nu11Writab1e;importorg.apache.hadoop.io.Text;importorg.apache.hadoop.napreduce.Reducer;importjava.io.IOException；pub1icc1assDupReducerextendsReducer<Text,Nu11Writab1e,Text,Nu11writab1e>Overrideprotectedvoidreduce(Textkey,Iterab1e<Nu11Writab1e>va1ues,Contextcontext)throwsIOExceptionjInterruptedEcontext.write(newText(key),Nu11Writab1e.get();)方法2：12346789111121415161718192021221.2 设计思路数据去重的最终目标是让原始数据中出现次数超过一次三数据在输出文件中只出现一次。Ma陶入key为行号，Va1Ue为行的内容