本文共 16010 字，大约阅读时间需要 53 分钟。
SecureCRT8.0
VMware12 RHEL 6.6 x64hadoop0 192.168.248.150
hadoop1 192.168.248.151 hadoop2 192.168.248.152 hadoop3 192.168.248.153sudo vim /etc/sysconfig/network-scripts/ifcfg-eth0
IPADDR=192.168.248.15?如果网络不行（由于虚拟机克隆）
检查/etc/udev/rule.d/70-persistent-net.rules ifcfg-eth0中的HWADDR设置hostname：
vim /etc/sysconfig/network HOSTNAME=hadoop?service iptables stop
vim /etc/selinux/config
SELINUX=disabledsetenforce 0
useradd hadoop
passwd hadoop #hadoopvim /etc/hosts
192.168.248.150 hadoop0 192.168.248.151 hadoop1 192.168.248.152 hadoop2 192.168.248.153 hadoop3生成非对称秘钥
su - hadoop ssh-keygen -t rsa拷贝NameNode公钥到其他节点上
[hadoop@hadoop0 ~]$ cp id_rsa.pub authorized_keys scp ~/.ssh/id_rsa.pub hadoop1:~/.ssh/authorized_keys scp ~/.ssh/id_rsa.pub hadoop2:~/.ssh/authorized_keys scp ~/.ssh/id_rsa.pub hadoop3:~/.ssh/authorized_keys如果不能ssh免密登录，则其它节点上
chmod 600 authorized_keyssftp上传
sftp> ls sftp> lcd D:\归档\软件\笔记本\linux sftp> lls jdk-8u40-linux-x64.tar.gz web sftp> put jdk-8u40-linux-x64.tar.gzscp jdk-8u40-linux-x64.tar.gz hadoop?:~/
tar zxf jdk-8u40-linux-x64.tar.gz
su - root
mv /home/hadoop/jdk1.8.0_40/ /opt/vim /etc/profile
export JAVA_HOME=/opt/jdk1.8.0_40 export JRE_HOME=/opt/jdk1.8.0_40/jre export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH 配置生效：
source /etc/profile验证：
java -version先配置一个节点（master，再复制到其它slave中）
sftp> put hadoop-2.6.4.tar.gz[hadoop@hadoop0 ~]$ tar zxf hadoop-2.6.4.tar.gz
vim /etc/profile
export HADOOP_HOME=/home/hadoop/hadoop-2.6.4 export PATH=$HADOOP_HOME/bin:$PATH 配置文件放在$HADOOP_HOME/etc/hadoop/下
vim hadoop-env.sh export JAVA_HOME=/opt/jdk1.8.0_40 vim yarn-env.sh
export JAVA_HOME=/opt/jdk1.8.0_40 vim slaves (这里没有了master配置文件)
hadoop1 hadoop2 hadoop3vim core-site.xml
hadoop.tmp.dir /home/hadoop/hadoop/tmp A base for other temporary directories. fs.defaultFS hdfs://hadoop0:9000 io.file.buffer.size 4096
vim hdfs-site.xml
dfs.namenode.name.dir /home/hadoop/hadoop/name dfs.datanode.data.dir /home/hadoop/hadoop/data dfs.replication 2 dfs.nameservices hadoop0 dfs.namenode.secondary.http-address hadoop0:50090 dfs.webhdfs.enabled true dfs.permissions false
cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
mapreduce.framework.name yarn true mapreduce.jobtracker.http.address hadoop0:50030 mapreduce.jobhistory.address hadoop0:10020 mapreduce.jobhistory.webapp.address hadoop0:19888 mapred.job.tracker hadoop0:9001
vim yarn-site.xml
yarn.resourcemanager.hostname hadoop0 yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.address hadoop0:8032 yarn.resourcemanager.scheduler.address hadoop0:8030 yarn.resourcemanager.resource-tracker.address hadoop0:8031 yarn.resourcemanager.admin.address hadoop0:8033 yarn.resourcemanager.webapp.address hadoop0:8088 yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler
复制到其它节点中
[hadoop@hadoop0 ~]$ scp -r hadoop-2.6.4 hadoop?:~/格式化文件系统
namenode格式化文件系统 $hadoop namenode -format hadoop命令在~/hadoop-2.6.2/bin中启动进程
[hadoop@hadoop0 hadoop-2.6.4]$ sbin/hadoop-daemon.sh start namenode[hadoop@hadoop0 hadoop-2.6.4]$ sbin/hadoop-daemon.sh start datanode
或者一次性启动
[hadoop@hadoop0 hadoop-2.6.4]$ sbin/start-all.sh检查节点配置情况
hadoop dfsadmin -report 错误情况： 可能配置tag错误； 拒绝连接，关闭防火墙[hadoop@hadoop0 hadoop-2.6.4]$ jps
4272 ResourceManager 4032 NameNode 4534 Jps关闭
[hadoop@hadoop0 hadoop-2.6.4]$ sbin/stop-all.sh网页节点管理
http://192.168.248.150:8088/cluster 网页资源管理
http://192.168.248.150:50070/dfshealth.html#tab-overview 参考：
Apache官网上提供的Hadoop本地库是32位的，如果我们的Linux服务器是64位的话，就会现问题。
我们在64位服务器执行Hadoop命令时，则会报以下错误：WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-Java classes where applicable
为了解决上述问题，我们就需要自己编译一个64位的hadoop版本。
yum install gcc
yum install cmake yum install gcc-c++wget
安装maven： 参考：tar xvf apache-maven-3.3.9-bin.tar.gzmv apache-maven-3.3.9/ /usr/local/vim /etc/profileexport MAVEN_HOME=/usr/local/apache-maven-3.3.9export PATH=$PATH:$MAVEN_HOME/binsource /etc/profile[root@hadoop0 hadoop]# mvn -vApache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)Maven home: /usr/local/apache-maven-3.3.9Java version: 1.8.0_40, vendor: Oracle CorporationJava home: /opt/jdk1.8.0_40/jreDefault locale: zh_CN, platform encoding: UTF-8OS name: "linux", version: "2.6.32-504.el6.x86_64", arch: "amd64", family: "unix"
要求版本protobuf-2.5.0
wgetcd protobuf-2.5.0/
./configure -prefix=/usr/local/protobuf-2.5.0 make && make installvim /etc/profile
export PROTOBUF=/usr/local/protobuf-2.5.0 export PATH=PROTOBUF/bin:PATHprotoc –version
wget
tar zxf apache-ant-1.9.7-bin.tar.gzmv apache-ant-1.9.7/ /usr/local/vim /etc/profileexport ANT_HOME=/usr/local/apache-ant-1.9.7export PATH=$PATH:$ANT_HOME/bin source /etc/profileyum install autoconf automake libtoolyum install openssl-devel
安装findbugs
vim /etc/profileexport FINDBUGS_HOME=/usr/local/findbugs-3.0.1/export PATH=$FINDBUGS_HOME/bin:$PATH
wget
tar zxf hadoop-2.6.4-src.tar.gzcd hadoop-2.6.4-src/
more BUILDING.txt
查看如何编译安装mvn package -DskipTests -Pdist,native,docs，其中docs根据需要添加。
mvn clean package -Pdist,native -DskipTests -Dtar 编译过程中，需要下载很多包，等待时间比较长。当看到hadoop各个项目都编译成功,即出现一系列的SUCCESS之后，即为编译成功。 编译大概需要二三十分钟，看到一堆的SUCCESS，说明编译成功，编译后的项目在hadoop-2.6.0-src/hadoop-dist/target/hadoop-2.6.0中。
只需要将编译后项目中的/hadoop-dist/lib/native文件夹覆盖至之前32位的项目中即可（当然整个项目覆盖也可以），并在配置文件$HADOOP_HOME/etc/hadoop/hadoop-env.sh末尾添加：export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native"。
编译过程中需要下载安装包，有时候可能由于网络的原因，导致安装包下载不完整，而出现编译错误。
Remote host closed connection during handshake: SSL peer shut down incorrectly.......
解决方案:需要重新新多编译几次即可通过。
mvn package -Pdist,native -DskipTests -Dtar
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /home/hadoop/hadoop-2.6.4-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: 错误: 意外的结束标记: [ERROR] * [ERROR] ^ [ERROR] [ERROR] Command line was: /opt/jdk1.8.0_40/jre/../bin/javadoc @options @packages [ERROR] [ERROR] Refer to the generated Javadoc files in ‘/home/hadoop/hadoop-2.6.4-src/hadoop-common-project/hadoop-annotations/target’ dir. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-annotations更换jdk1.7
tar zxf jdk-7u80-linux-x64.gz mv jdk1.7.0_80/ /opt/ vim /etc/profile[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.4:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is ‘libprotoc 2.6.1’, expected version is ‘2.5.0’ -
protocbuf版本不对，下2.5.0
wget cd protobuf-2.5.0 ./configure –prefix=/usr/local/protobuf-2.5.0/ make make install vim /etc/profile export PATH=$PATH:/usr/local/protobuf-2.5.0/bin[ERROR] Failed to execute goal org.codehaus.mojo.jspc:jspc-maven-plugin:2.0-alpha-3:compile (hdfs) on project hadoop-hdfs: Execution hdfs of goal org.codehaus.mojo.jspc:jspc-maven-plugin:2.0-alpha-3:compile failed: Plugin org.codehaus.mojo.jspc:jspc-maven-plugin:2.0-alpha-3 or one of its dependencies could not be resolved: The following artifacts could not be resolved: tomcat:jasper-compiler-jdt:jar:5.5.15, org.eclipse.jdt:core:jar:3.1.1: Could not transfer artifact tomcat:jasper-compiler-jdt:jar:5.5.15 from/to central (): GET request of: tomcat/jasper-compiler-jdt/5.5.15/jasper-compiler-jdt-5.5.15.jar from central failed: Connection reset -> [Help 1]
mvn包缺少，可以自己到
下载，放入~/.m2/指定目录mvn package -Pdist,native -DskipTests -Dtar -e -X
编译完成
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [04:57 min][INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 7.926 s][INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 11.552 s][INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 37.191 s][INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 11.227 s][INFO] hadoop-yarn-client ................................. SUCCESS [ 11.344 s][INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.236 s][INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 6.616 s][INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 3.877 s][INFO] hadoop-yarn-site ................................... SUCCESS [ 0.171 s][INFO] hadoop-yarn-registry ............................... SUCCESS [ 10.513 s][INFO] hadoop-yarn-project ................................ SUCCESS [ 10.549 s][INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.274 s][INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 37.510 s][INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 26.697 s][INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 8.445 s][INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 16.746 s][INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 14.682 s][INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [02:22 min][INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 4.852 s][INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 9.098 s][INFO] hadoop-mapreduce ................................... SUCCESS [ 8.473 s][INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 44.417 s][INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 56.786 s][INFO] Apache Hadoop Archives ............................. SUCCESS [ 5.743 s][INFO] Apache Hadoop Rumen ................................ SUCCESS [ 9.956 s][INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 8.721 s][INFO] Apache Hadoop Data Join ............................ SUCCESS [ 6.602 s][INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 4.001 s][INFO] Apache Hadoop Extras ............................... SUCCESS [ 6.550 s][INFO] Apache Hadoop Pipes ................................ SUCCESS [ 16.753 s][INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 9.256 s][INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:50 min][INFO] Apache Hadoop Client ............................... SUCCESS [ 15.419 s][INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.856 s][INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 10.593 s][INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 22.169 s][INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.118 s][INFO] Apache Hadoop Distribution ......................... SUCCESS [01:20 min][INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 03:01 h[INFO] Finished at: 2016-07-23T11:25:52+08:00[INFO] Final Memory: 90M/237M[INFO] ------------------------------------------------------------------------
在hadoop-dist/target/ 已经生成了可执行文件，再次执行之前的配置hadoop的方法即可
复制之前的配置的时候用nano编辑[hadoop@hadoop0 hadoop-2.6.4]$ sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [hadoop0]hadoop0: starting namenode, logging to /home/hadoop/hadoop-2.6.4/logs/hadoop-hadoop-namenode-hadoop0.outhadoop2: starting datanode, logging to /home/hadoop/hadoop-2.6.4/logs/hadoop-hadoop-datanode-hadoop2.outhadoop1: starting datanode, logging to /home/hadoop/hadoop-2.6.4/logs/hadoop-hadoop-datanode-hadoop1.outhadoop3: starting datanode, logging to /home/hadoop/hadoop-2.6.4/logs/hadoop-hadoop-datanode-hadoop3.outstarting yarn daemonsstarting resourcemanager, logging to /home/hadoop/hadoop-2.6.4/logs/yarn-hadoop-resourcemanager-hadoop0.outhadoop3: starting nodemanager, logging to /home/hadoop/hadoop-2.6.4/logs/yarn-hadoop-nodemanager-hadoop3.outhadoop2: starting nodemanager, logging to /home/hadoop/hadoop-2.6.4/logs/yarn-hadoop-nodemanager-hadoop2.outhadoop1: starting nodemanager, logging to /home/hadoop/hadoop-2.6.4/logs/yarn-hadoop-nodemanager-hadoop1.out
下载hadoop-eclipse-plugin-2.6.4.jar
放入hadoop/plugins/解压缩hadoop-2.6.4.tar.gz到D:\develop\hadoop-2.6.4，在eclipse的Windows->Preferences的Hadoop Map/Reduce中设置安装目录。
打开Windows->Open Perspective中的Map/Reduce，在此perspective下进行hadoop程序开发。
打开Windows->Show View中的Map/Reduce Locations，选择New Hadoop location…新建hadoop连接。
Map/Reduce(V2)Master和DFS Master的Host、Port分别为： mapred-site.xml和core-site.xml中配置的地址及端口配置Advanced parameters
设置参数hadoop.tmp.dir为core-site.xml中的配置An internal error occurred during: “Map/Reduce location status updater”.
java.lang.NullPointerException“NullPointerException” is just Because you don’t have any directory in your hdfs.
Try that to build a directory:hdfs dfs -mkdir -p input
hadoop fs -put ./myTest*.txt input
转载地址：http://tnzxx.baihongyu.com/