2.搭建Hadoop HA

发表于 | 更新于 | 分类于

准备工作

  • 1.准备好3台虚拟机
  • 2.3台虚拟机互信
  • 3.准备好安装包
    • jdk-8u161-linux-x64.tar.gz
    • hadoop-2.6.0-cdh5.7.0.tar.gz
    • zookeeper-3.4.12.tar.gz
  • 4.Xshell5

新增用户hadoop

在3台虚拟机上面都新建一个hadoop用户

1
# useradd hadoop

在3台虚拟机上面都新建目录

1
2
# su - hadoop
$ mkdir app source software data tmp

安装JDK

创建jdk存放路径

1
2
# mkdir -p /usr/java
# cd /usr/java

上传jdk压缩包

1
# rz

解压jdk压缩包

1
2
# tar -xzvf jdk-8u161-linux-x64.tar.gz
# rm -f jdk-8u161-linux-x64.tar.gz

传输到其他两台

1
2
# scp -r jdk1.8.0_161 root@hadoop002:/usr/java
# scp -r jdk1.8.0_161 root@hadoop003:/usr/java

修改三台机子上的jdk的持有者

1
# chown -R root:root /usr/java

配置JDK环境变量

1
# vi /etc/profile

在文件的末尾加入，以下语句，并保存

1
2
3
#env
export JAVA_HOME=/usr/java/jdk1.8.0_161
export PATH=$JAVA_HOME/bin:$PATH

使配置生效

1
# source /etc/profile

查看jdk是否成功安装

1
# java -version

在另外两台机子也一样做一遍，至此我们的jdk就安装完成了！

集群规划

IP HOST 安装软件 进程
192.168.137.190 hadoop001 Hadoop、Zookeeper NameNode
DFSZKFailoverController
JournalNode
DataNode
ResourceManager
JobHistoryServer
NodeManager
QuorumPeerMain
192.168.137.191 hadoop002 Hadoop、Zookeeper NameNode
DFSZKFailoverController
JournalNode
DataNode
ResourceManager
NodeManager
QuorumPeerMain
192.168.137.192 hadoop003 Hadoop、Zookeeper JournalNode
DataNode
QuorumPeerMain
NodeManager

安装ZK

上传压缩包

上传ZK的压缩文件文件到hadoop001的software里面

1
2
$ cd software
$ rz

传给另外两台机子

1
2
$ scp ~/software/zookeeper-3.4.12.tar.gz hadoop@hadoop002:/home/hadoop/software/
$ scp ~/software/zookeeper-3.4.12.tar.gz hadoop@hadoop003:/home/hadoop/software/

在所有机子上一起运行解压ZK压缩文件到app文件夹里面

1
2
$ cd ~/software
$ tar -xzvf zookeeper-3.4.12.tar.gz -C ../app/

配置ZK_HOME

1
$ vi ~/.bash_profile
1
2
3
4
5
6
7
8
9
10
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export ZK_HOME=/home/hadoop/app/zookeeper-3.4.12
export PATH=$ZK_HOME/bin:$PATH
1
$ source ~/.bash_profile

修改配置

1
2
$ cd ~/app/zookeeper-3.4.12/conf
$ cp zoo_sample.cfg zoo.cfg

1
$ vi zoo.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/home/hadoop/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=hadoop001:2888:3888
server.2=hadoop002:2888:3888
server.3=hadoop003:2888:3888

传给其他节点

1
2
$ scp zoo.cfg hadoop@hadoop002:/home/hadoop/app/zookeeper-3.4.12/conf/
$ scp zoo.cfg hadoop@hadoop002:/home/hadoop/app/zookeeper-3.4.12/conf/
在三个节点上

切换到data目录,并创建一个zookeeper目录

1
2
$ cd ~/data
$ mkdir zookeeper

在zookeeper目录里面，创建一个myid文件

1
$ touch myid
各个节点分开做

然后在hadoop001节点上，在myid文件写入1

1
$ echo 1 >myid

在hadoop002节点上，在myid文件写入2

1
$ echo 2 >myid

在hadoop003节点上，在myid文件写入3

1
$ echo 3 >myid

启动集群

1
$ zkServer.sh start

安装Hadoop

上传hadoop的压缩包

在hadoop001机子上使用hadoop用户在software目录上传hadoop的压缩包

1
2
$ cd ~/software
$ rz

拷贝到其他两台机子上

1
2
$ scp hadoop-2.6.0-cdh5.7.0.tar.gz hadoop@hadoop002:/home/hadoop/software/
$ scp hadoop-2.6.0-cdh5.7.0.tar.gz hadoop@hadoop003:/home/hadoop/software/

解压缩包

三台自己一起执行

1
2
$ cd ~/software
$ tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C ../app

配置HADOOP_HOME

在hadoop001上面配置一下HADOOP_HOME

1
$ vi ~/.bash_profile
1
2
3
4
5
6
7
8
9
10
11
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export ZK_HOME=/home/hadoop/app/zookeeper-3.4.12
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZK_HOME/bin:$PATH
1
$ source ~/.bash_profile

修改hadoop-env.sh

1
$ vi /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/hadoop-env.sh

修改core-site.xml

1
$ vi /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/core-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!--Yarn 需要使用 fs.defaultFS 指定NameNode URI -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://zzfHadoop</value>
    </property>
    <!-- ==============================Trash机制======================================= -->
    <property>
    <!--多长时间创建CheckPoint NameNode截点上运行的CheckPointer 从Current文件夹创建CheckPoint;默认：0 由fs.trash.interval项指定 -->
        <name>fs.trash.checkpoint.interval</name>
        <value>0</value>
    </property>
    <property>
    <!--多少分钟.Trash下的CheckPoint目录会被删除,该配置服务器设置优先级大于客户端，默认：0 不删除 -->
        <name>fs.trash.interval</name>
        <value>1440</value>
    </property>
    <!--指定hadoop临时目录, hadoop.tmp.dir 是hadoop文件系统依赖的基础配置，很多路径都依赖它。如果hdfs-site.xml中不>配 置namenode和datanode的存放位置，默认就放在这>个路径中 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop/tmp</value>
    </property>
    <!-- 指定zookeeper地址 -->
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop001:2181,hadoop002:2181,hadoop003:2181</value>
    </property>
    <!--指定ZooKeeper超时间隔，单位毫秒 -->
    <property>
        <name>ha.zookeeper.session-timeout.ms</name>
        <value>2000</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hadoop.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hadoop.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>io.compression.codecs</name>
        <value>org.apache.hadoop.io.compress.GzipCodec,
          org.apache.hadoop.io.compress.DefaultCodec,
          org.apache.hadoop.io.compress.BZip2Codec,
          org.apache.hadoop.io.compress.SnappyCodec</value>
    </property>
</configuration>

修改hdfs-site.xml

1
2
$ mkdir -p /home/hadoop/data/dfs/name /home/hadoop/data/dfs/data
$ vi /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/hdfs-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!--HDFS超级用户 -->
	<property>
		<name>dfs.permissions.superusergroup</name>
		<value>hadoop</value>
	</property>

	<!--开启web hdfs -->
	<property>
		<name>dfs.webhdfs.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>/home/hadoop/data/dfs/name</value>
		<description> namenode 存放name table(fsimage)本地目录（需要修改）</description>
	</property>
	<property>
		<name>dfs.namenode.edits.dir</name>
		<value>${dfs.namenode.name.dir}</value>
		<description>namenode存放 transaction file(edits)本地目录（需要修改）</description>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>/home/hadoop/data/dfs/data</value>
		<description>datanode存放block本地目录（需要修改）</description>
	</property>
	<property>
		<name>dfs.replication</name>
		<value>3</value>
	</property>
	<!-- 块大小256M （默认128M） -->
	<property>
		<name>dfs.blocksize</name>
		<value>268435456</value>
	</property>
	<!--======================================================================= -->
	<!--HDFS高可用配置 -->
	<!--指定hdfs的nameservice为zzfHadoop,需要和core-site.xml中的保持一致 -->
	<property>
		<name>dfs.nameservices</name>
		<value>zzfHadoop</value>
	</property>
	<property>
		<!--设置NameNode IDs 此版本最大只支持两个NameNode -->
		<name>dfs.ha.namenodes.zzfHadoop</name>
		<value>nn1,nn2</value>
	</property>

	<!-- Hdfs HA: dfs.namenode.rpc-address.[nameservice ID] rpc 通信地址 -->
	<property>
		<name>dfs.namenode.rpc-address.zzfHadoop.nn1</name>
		<value>hadoop001:8020</value>
	</property>
	<property>
		<name>dfs.namenode.rpc-address.zzfHadoop.nn2</name>
		<value>hadoop002:8020</value>
	</property>

	<!-- Hdfs HA: dfs.namenode.http-address.[nameservice ID] http 通信地址 -->
	<property>
		<name>dfs.namenode.http-address.zzfHadoop.nn1</name>
		<value>hadoop001:50070</value>
	</property>
	<property>
		<name>dfs.namenode.http-address.zzfHadoop.nn2</name>
		<value>hadoop002:50070</value>
	</property>

	<!--==================Namenode editlog同步 ============================================ -->
	<!--保证数据恢复 -->
	<property>
		<name>dfs.journalnode.http-address</name>
		<value>0.0.0.0:8480</value>
	</property>
	<property>
		<name>dfs.journalnode.rpc-address</name>
		<value>0.0.0.0:8485</value>
	</property>
	<property>
		<!--设置JournalNode服务器地址，QuorumJournalManager 用于存储editlog -->
		<!--格式：qjournal://<host1:port1>;<host2:port2>;<host3:port3>/<journalId> 端口同journalnode.rpc-address -->
		<name>dfs.namenode.shared.edits.dir</name>
		<value>qjournal://hadoop001:8485;hadoop002:8485;hadoop003:8485/zzfHadoop</value>
	</property>

	<property>
		<!--JournalNode存放数据地址 -->
		<name>dfs.journalnode.edits.dir</name>
		<value>/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/data/dfs/jn</value>
	</property>
	<!--==================DataNode editlog同步 ============================================ -->
	<property>
		<!--DataNode,Client连接Namenode识别选择Active NameNode策略 -->
                             <!-- 配置失败自动切换实现方式 -->
		<name>dfs.client.failover.proxy.provider.zzfHadoop</name>
		<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
	</property>
	<!--==================Namenode fencing：=============================================== -->
	<!--Failover后防止停掉的Namenode启动，造成两个服务 -->
	<property>
		<name>dfs.ha.fencing.methods</name>
		<value>sshfence</value>
	</property>
	<property>
		<name>dfs.ha.fencing.ssh.private-key-files</name>
		<value>/home/hadoop/.ssh/id_rsa</value>
	</property>
	<property>
		<!--多少milliseconds 认为fencing失败 -->
		<name>dfs.ha.fencing.ssh.connect-timeout</name>
		<value>30000</value>
	</property>

	<!--==================NameNode auto failover base ZKFC and Zookeeper====================== -->
	<!--开启基于Zookeeper  -->
	<property>
		<name>dfs.ha.automatic-failover.enabled</name>
		<value>true</value>
	</property>
	<!--动态许可datanode连接namenode列表 -->
	 <property>
	   <name>dfs.hosts</name>
	   <value>/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/slaves</value>
	 </property>
</configuration>

修改/mapred-site.xml

1
2
3
$ cd ~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
$ cp mapred-site.xml.template mapred-site.xml
$ vi mapred-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!-- 配置 MapReduce Applications -->
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
	<!-- JobHistory Server ============================================================== -->
	<!-- 配置 MapReduce JobHistory Server 地址 ，默认端口10020 -->
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>hadoop001:10020</value>
	</property>
	<!-- 配置 MapReduce JobHistory Server web ui 地址， 默认端口19888 -->
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>hadoop001:19888</value>
	</property>

<!-- 配置 Map段输出的压缩,snappy-->
  <property>
      <name>mapreduce.map.output.compress</name> 
      <value>true</value>
  </property>
              
  <property>
      <name>mapreduce.map.output.compress.codec</name> 
      <value>org.apache.hadoop.io.compress.SnappyCodec</value>
   </property>

</configuration>

修改yarn-site.xml

1
$ vi /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/yarn-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!-- nodemanager 配置 ================================================= -->
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
	</property>
	<property>
		<name>yarn.nodemanager.localizer.address</name>
		<value>0.0.0.0:23344</value>
		<description>Address where the localizer IPC is.</description>
	</property>
	<property>
		<name>yarn.nodemanager.webapp.address</name>
		<value>0.0.0.0:23999</value>
		<description>NM Webapp address.</description>
	</property>

	<!-- HA 配置 =============================================================== -->
	<!-- Resource Manager Configs -->
	<property>
		<name>yarn.resourcemanager.connect.retry-interval.ms</name>
		<value>2000</value>
	</property>
	<property>
		<name>yarn.resourcemanager.ha.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
		<value>true</value>
	</property>
	<!-- 使嵌入式自动故障转移。HA环境启动，与 ZKRMStateStore 配合 处理fencing -->
	<property>
		<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
		<value>true</value>
	</property>
	<!-- 集群名称，确保HA选举时对应的集群 -->
	<property>
		<name>yarn.resourcemanager.cluster-id</name>
		<value>yarn-cluster</value>
	</property>
	<property>
		<name>yarn.resourcemanager.ha.rm-ids</name>
		<value>rm1,rm2</value>
	</property>


    <!--这里RM主备结点需要单独指定,（可选）
	<property>
		 <name>yarn.resourcemanager.ha.id</name>
		 <value>rm2</value>
	 </property>
	 -->

	<property>
		<name>yarn.resourcemanager.scheduler.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
	</property>
	<property>
		<name>yarn.resourcemanager.recovery.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
		<value>5000</value>
	</property>
	<!-- ZKRMStateStore 配置 -->
	<property>
		<name>yarn.resourcemanager.store.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
	</property>
	<property>
		<name>yarn.resourcemanager.zk-address</name>
		<value>hadoop001:2181,hadoop002:2181,hadoop003:2181</value>
	</property>
	<property>
		<name>yarn.resourcemanager.zk.state-store.address</name>
		<value>hadoop001:2181,hadoop002:2181,hadoop003:2181</value>
	</property>
	<!-- Client访问RM的RPC地址 (applications manager interface) -->
	<property>
		<name>yarn.resourcemanager.address.rm1</name>
		<value>hadoop001:23140</value>
	</property>
	<property>
		<name>yarn.resourcemanager.address.rm2</name>
		<value>hadoop002:23140</value>
	</property>
	<!-- AM访问RM的RPC地址(scheduler interface) -->
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm1</name>
		<value>hadoop001:23130</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm2</name>
		<value>hadoop002:23130</value>
	</property>
	<!-- RM admin interface -->
	<property>
		<name>yarn.resourcemanager.admin.address.rm1</name>
		<value>hadoop001:23141</value>
	</property>
	<property>
		<name>yarn.resourcemanager.admin.address.rm2</name>
		<value>hadoop002:23141</value>
	</property>
	<!--NM访问RM的RPC端口 -->
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
		<value>hadoop001:23125</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
		<value>hadoop002:23125</value>
	</property>
	<!-- RM web application 地址 -->
	<property>
		<name>yarn.resourcemanager.webapp.address.rm1</name>
		<value>hadoop001:8088</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.address.rm2</name>
		<value>hadoop002:8088</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.https.address.rm1</name>
		<value>hadoop001:23189</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.https.address.rm2</name>
		<value>hadoop002:23189</value>
	</property>

	<property>
	   <name>yarn.log-aggregation-enable</name>
	   <value>true</value>
	</property>
	<property>
		 <name>yarn.log.server.url</name>
		 <value>http://hadoop001:19888/jobhistory/logs</value>
	</property>


	<property>
		<name>yarn.nodemanager.resource.memory-mb</name>
		<value>2048</value>
	</property>
	<property>
		<name>yarn.scheduler.minimum-allocation-mb</name>
		<value>1024</value>
		<discription>单个任务可申请最少内存，默认1024MB</discription>
	 </property>

  
  <property>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>2048</value>
	<discription>单个任务可申请最大内存，默认8192MB</discription>
  </property>

   <property>
       <name>yarn.nodemanager.resource.cpu-vcores</name>
       <value>2</value>
    </property>
</configuration>

修改 slaves

1
[hadoop@hadoop001 hadoop]$ vi slaves
1
2
3
hadoop001
hadoop002
hadoop003

将配置文件传送到其他节点

1
2
3
$ cd ~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/
$ scp * hadoop@hadoop002:/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/
$ scp * hadoop@hadoop003:/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/

启动JN集群

1
2
$ hadoop-daemon.sh start journalnode
$ jps

格式化NameNode

在hadoop001节点上输入以下命令

1
$ hadoop namenode -format
1
2
3
4
5
6
7
8
9
10
11
12
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
..........
..........
18/11/27 17:33:34 INFO namenode.FSImage: Allocated new BlockPoolId: BP-793957562-192.168.137.190-1543368814407
18/11/27 17:33:34 INFO common.Storage: Storage directory /home/hadoop/data/dfs/name has been successfully formatted.
18/11/27 17:33:34 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/11/27 17:33:34 INFO util.ExitUtil: Exiting with status 0
18/11/27 17:33:34 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop001/192.168.137.190
************************************************************/

同步到第二个NN的目录下

1
$ scp -r /home/hadoop/data/dfs/* hadoop@hadoop002:/home/hadoop/data/dfs/

初始化ZKFC

在hadoop001节点上：

1
$ hdfs zkfc -formatZK

启动HDFS集群

在hadoop001节点上

1
$  start-dfs.sh

检查一下

1
$ jps| grep -v Jps

启动YARN

在hadoop001节点

1
$ start-yarn.sh

在hadoop002节点启动RN(standby)

1
$ yarn-daemon.sh start resourcemanager

检查一下

1
$ jps| grep -v Jps

至此我们集群的搭建就完成了。

关闭集群

在hadoop001上

1
$ stop-yarn.sh

在hadoop002上

1
$ yarn-daemon.sh stop resourcemanager

在hadoop001上

1
$ stop-dfs.sh

在三个节点上

1
$  zkServer.sh stop

再次启动集群

在三个节点上

1
$ zkServer.sh start

在hadoop001上

1
$ start-dfs.sh

在hadoop001上

1
$ start-yarn.sh

在hadoop002上

1
$ yarn-daemon.sh start resourcemanager

监控集群

在hadoop001上

1
$ hdfs dfsadmin -report

开启步骤整理

  • ZK
  • NN
  • DN
  • JN
  • ZKFC
  • YARN RM(active) NN
  • YARN RM(standby)

关闭步骤整理

  • YARN RM(active) NN
  • YARN RM(standby)
  • namenode
  • datanode
  • journalnode
  • zkfc
  • ZK