之前写过一些如何安装Cloudera Hadoop的文章，安装hadoop过程中，最开始是手动安装apache版本的hadoop，其次是使用Intel的IDH管理界面安装IDH的hadoop，再然后分别手动和通过cloudera manager安装hadoop，也使用bigtop-util yum方式安装过apache的hadoop。

安装过程中参考了很多网上的文章，解压缩过cloudera的cloudera-manager-installer.bin，发现并修复了IDH shell脚本中关于puppt的自认为是bug的一个bug，最后整理出了一个自动安装hadoop的shell脚本，脚本托管在github上面: hadoop-install。

hadoop安装文章

博客中所有关于安装hadoop的文章列出如下：

【笔记】Hadoop安装部署

手动安装Cloudera Hive CDH

手动安装Cloudera HBase CDH

手动安装Cloudera Hadoop CDH

安装impala过程

从yum安装Cloudera CDH集群

通过Cloudera Manager安装CDH

hadoop-install

hadoop-install上脚本，all-in-one-install.sh是在一个节点上安装hdfs、hive、yarn、zookeeper和hbase，编写该脚本是为了在本机（fedora19系统）上调试mapreduce、hive和hbase；cluster-install.sh是在多个节点上安装hadoop集群，同样目前完成了hdfs、hive、yarn、zookeeper和hbase的自动安装。

脚本片段

IDH安装脚本中有一些写的比较好的shell代码片段，摘出如下，供大家学习。

检测操作系统版本

1
2
3
4
5
6
7
8
9
10
11
12
1( grep -i "CentOS" /etc/issue > /dev/null ) && OS_DISTRIBUTOR=centos

2( grep -i "Red[[:blank:]]*Hat[[:blank:]]*Enterprise[[:blank:]]*Linux" /etc/issue > /dev/null ) && OS_DISTRIBUTOR=rhel

3( grep -i "Oracle[[:blank:]]*Linux" /etc/issue > /dev/null ) && OS_DISTRIBUTOR=oel

4( grep -i "Asianux[[:blank:]]*Server" /etc/issue > /dev/null ) && OS_DISTRIBUTOR=an

5( grep -i "SUSE[[:blank:]]*Linux[[:blank:]]*Enterprise[[:blank:]]*Server" /etc/issue > /dev/null ) && OS_DISTRIBUTOR=sles

6( grep -i "Fedora" /etc/issue > /dev/null ) && OS_DISTRIBUTOR=fedora

7

8major_revision=`grep -oP '\d+' /etc/issue | sed -n "1,1p"`

9minor_revision=`grep -oP '\d+' /etc/issue | sed -n "2,2p"`

10OS_RELEASE="$major_revision.$minor_revision"

11

12

修改root密码

1
2
3
1echo 'redhat'|passwd root --stdin

2

3

修改dns

1
2
3
4
5
6
7
1# Set up nameservers.

2# http://ithelpblog.com/os/linux/redhat/centos-redhat/howto-fix-couldnt-resolve-host-on-centos-redhat-rhel-fedora/

3# http://stackoverflow.com/a/850731/1486325

4echo "nameserver 8.8.8.8" | tee -a /etc/resolv.conf

5echo "nameserver 8.8.4.4" | tee -a /etc/resolv.conf

6

7

修改操作系统时区

1
2
3
1cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

2

3

修改hosts文件

1
2
3
4
5
6
7
8
9
1cat > /etc/hosts <<EOF

2127.0.0.1       localhost

3

4192.168.56.121 cdh1

5192.168.56.122 cdh2

6192.168.56.123 cdh3

7EOF

8

9

去掉b文件中包括a文件的内容

1
2
3
1grep -vf a b >result.log

2

3

修改file-max

1
2
3
4
5
6
7
8
9
1echo -e "Global file limit ..."

2rst=`grep "^fs.file-max" /etc/sysctl.conf`

3if [ "x$rst" = "x" ] ; then

4    echo "fs.file-max = 727680" >> /etc/sysctl.conf || exit $?

5else

6    sed -i "s:^fs.file-max.*:fs.file-max = 727680:g" /etc/sysctl.conf

7fi

8

9

生成ssh公要

1
2
3
4
1[ ! -d ~/.ssh ] && ( mkdir ~/.ssh ) && ( chmod 600 ~/.ssh )

2yes|ssh-keygen -f ~/.ssh/id_rsa -t rsa -N "" && ( chmod 600 ~/.ssh/id_rsa.pub )

3

4

ssh设置无密码登陆

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1set timeout 20

2

3set host [lindex $argv 0]

4set password [lindex $argv 1]

5set pubkey [exec cat /root/.ssh/id_rsa.pub]

6set localsh [exec cat ./config_ssh_local.sh]

7

8#spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$host

9spawn ssh root@$host "

10umask 022

11mkdir -p  /root/.ssh

12echo \'$pubkey\' > /root/.ssh/authorized_keys

13echo \'$localsh\' >  /root/.ssh/config_ssh_local.sh

14cd /root/.ssh/; sh config_ssh_local.sh

15"

16expect {

17    timeout exit

18    yes/no  {send "yes\r";exp_continue}

19    assword {send "$password\r"}

20}

21expect eof

22#interact

23

24

配置JAVA_HOME

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
1### JAVA_HOME ###

2if [ -f ~/.bashrc ] ; then

3    sed -i '/^export[[:space:]]\{1,\}JAVA_HOME[[:space:]]\{0,\}=/d' ~/.bashrc

4    sed -i '/^export[[:space:]]\{1,\}CLASSPATH[[:space:]]\{0,\}=/d' ~/.bashrc

5    sed -i '/^export[[:space:]]\{1,\}PATH[[:space:]]\{0,\}=/d' ~/.bashrc

6fi

7echo "" >>~/.bashrc

8echo "export JAVA_HOME=/usr/java/latest" >>~/.bashrc

9echo "export CLASSPATH=.:\$JAVA_HOME/lib/tools.jar:\$JAVA_HOME/lib/dt.jar">>~/.bashrc

10echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> ~/.bashrc

11

12alternatives --install /usr/bin/java java /usr/java/latest 5

13alternatives --set java /usr/java/latest 

14source ~/.bashrc

15

16

格式化集群

1
2
3
1su -s /bin/bash hdfs -c 'yes Y | hadoop namenode -format >> /tmp/format.log 2>&1'

2

3

创建hadoop目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
1su -s /bin/bash hdfs -c "hadoop fs -chmod a+rw /"

2while read dir user group perm

3do

4     su -s /bin/bash hdfs -c "hadoop fs -mkdir -R $dir && hadoop fs -chmod -R $perm $dir && hadoop fs -chown -R $user:$group $dir"

5     echo "."

6done << EOF

7/tmp hdfs hadoop 1777 

8/tmp/hadoop-yarn mapred mapred 777

9/var hdfs hadoop 755 

10/var/log yarn mapred 1775 

11/var/log/hadoop-yarn/apps yarn mapred 1777

12/hbase hbase hadoop 755

13/user hdfs hadoop 777

14/user/history mapred hadoop 1777

15/user/root root hadoop 777

16/user/hive hive hadoop 777

17EOF

18

19

hive中安装并初始化postgresql

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1yum install postgresql-server postgresql-jdbc -y >/dev/null

2chkconfig postgresql on

3rm -rf /var/lib/pgsql/data

4rm -rf /var/run/postgresql/.s.PGSQL.5432

5service postgresql initdb

6

7sed -i "s/max_connections = 100/max_connections = 600/" /var/lib/pgsql/data/postgresql.conf

8sed -i "s/#listen_addresses = 'localhost'/listen_addresses = '*'/" /var/lib/pgsql/data/postgresql.conf

9sed -i "s/shared_buffers = 32MB/shared_buffers = 256MB/" /var/lib/pgsql/data/postgresql.conf

10sed -i "s/127.0.0.1\/32/0.0.0.0\/0/" /var/lib/pgsql/data/pg_hba.conf

11

12sudo cat /var/lib/pgsql/data/postgresql.conf | grep -e listen -e standard_conforming_strings

13

14rm -rf /usr/lib/hive/lib/postgresql-jdbc.jar

15ln -s /usr/share/java/postgresql-jdbc.jar /usr/lib/hive/lib/postgresql-jdbc.jar

16

17su -c "cd ; /usr/bin/pg_ctl start -w -m fast -D /var/lib/pgsql/data" postgres

18su -c "cd ; /usr/bin/psql --command \"create user hiveuser with password 'redhat'; \" " postgres

19su -c "cd ; /usr/bin/psql --command \"CREATE DATABASE metastore owner=hiveuser;\" " postgres

20su -c "cd ; /usr/bin/psql --command \"GRANT ALL privileges ON DATABASE metastore TO hiveuser;\" " postgres

21su -c "cd ; /usr/bin/psql -U hiveuser -d metastore -f /usr/lib/hive/scripts/metastore/upgrade/postgres/hive-schema-0.10.0.postgres.sql" postgres

22su -c "cd ; /usr/bin/pg_ctl restart -w -m fast -D /var/lib/pgsql/data" postgres

23

24

总结

更多脚本，请关注github：hadoop-install，你可以下载、使用并修改其中代码！

—-EOF—–

转载： http://blog.javachen.com/2013/08/02/hadoop-install-script/