数据平台安装配置-Hive

Posted on   |   In

前提: Hadoop 提前安装完成, 相关依赖包已安装. Mysql 5.5+ 已安装.

Hive 安装

安装路径:

解压压缩文件到 ~/workspace 下

修改文件名 `mv apache-hive-1.2.1-bin hive-1.2.1`

修改环境变量

1
2
3
export HIVE_HOME=/home/hadoop/workspace/hive-1.2.1
export CLASSPATH=.:$HIVE_HOME/lib:$CLASSPATH
export PATH=$HIVE_HOME/bin:$PATH

配置生效:

1
source ~/.bashrc

修改 hive 配置

配置文件在 ~/workspace/hive-1.2.1/conf

1
2
3
4
5
6
$ ls  ~/workspace/hive-1.2.1/conf
    beeline-log4j.properties.template
    hive-default.xml.template
    hive-env.sh.template
    hive-exec-log4j.properties.template
    hive-log4j.properties.template

cd ~/workspace/hive-1.2.1/conf #进入 hive 配置目录

1
2
3
cp hive-default.xml.template hive-site.xml
cp hive-env.sh.template hive-env.sh
cp hive-log4j.properties.template hive-log4j.properties

Hive 环境配置

1
2
3
4
$ vim hive-env.sh
    HADOOP_HOME=$HADOOP_HOME
    export HIVE_CONF_DIR=${HIVE_HOME}/conf
    export HIVE_AUX_JARS_PATH=${HIVE_HOME}/lib

Hive 站点设置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
$ vim hive-site.xml
    <property>
        <name>hive.querylog.location</name>
        <value>/home/data/hive/log</value>
    </property>
    <property>
        <name>hive.exec.scratchdir</name>
        <value>hdfs://h16:9000/hive/warehouse/</value>
        <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
    </property>
    <property>
        <name>hive.exec.local.scratchdir</name>
        <value>/home/data/hive/iotmp</value>
        <description>Local scratch space for Hive jobs</description>
    </property>
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>hdfs://h16:9000/hive/warehouse</value>
        <description>location of default database for the warehouse</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNoExist=true</value>
        <description>JDBC connect string for a JDBC metastore</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>Driver class name for a JDBC metastore</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
        <description>Username to use against metastore database</description>
    </property>
    <property>
         <name>javax.jdo.option.ConnectionPassword</name>
         <value>**</value>
         <description>password to use against metastore database</description>
    </property>
    <property>
        <name>hive.metastore.uris</name>
        <value/>
        <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
    </property>

其中,
 hive.metastore.warehouse.dir  为数据文件目录
 hive.exec.scratchdir 为临时文件目录
 hive.querylog.location 为 hive 相关日志目录
 javax.jdo.option.ConnectionURL 为元数据连接字符串
 hive.exec.local.scratchdir 为本地的临时文件目录
 javax.jdo.option.ConnectionURL 为 元数据存储的数据库连接方式
 javax.jdo.option.ConnectionDriverName 为 mysql 驱动
 javax.jdo.option.ConnectionUserName 为 mysql 中的 hive 用户
 javax.jdo.option.ConnectionPassword 为 mysql 用户的密码
 hive.metastore.uris 为 thrift 地址, 用于通过 jdbc/odbc 连接 hive

ps:

上面的文件路径需要创建, 且修改权限.

1
2
3
hadoop fs -mkdir -p /hive/warehouse/
hadoop fs -chmod g+w /hive/warehouse/

因为要用 Mysql 作为 hive 元数据数据库, 下载 mysql-connector-java-5.1.37.tar.gz, 解压后将此
目录下的 mysql-connector-java-5.1.37-bin.jar 复制到 $HIVE_HOME/lib 下.

要通过 jdbc/odbc 连接 hive, 需要启动 metastore shift, 即配置 hive.metastore.uris 参数.

Hive 验证:

1
2
3
$ hive
>> show databases;
default

参考资料

Hive 安装步骤