Skip to content

Accessing Hadoop Data Using Hive

January 19, 2016

Writing map/reduce programs to analyze your Big Data can get complex. Hive can help make querying your data much easier

Fundamentals of Hive

Hive System Architecture

Hadoop was the solution for large data storage but using Hadoop was not easy task for end users, especially for those who were not familiar with the map reduce concept. Hive is an easy way to work with data stored in HDFS.

Basically Hive is SQL for Hadoop cluster. It is an open source data warehouse system on top of HDFS that adds structure to the data.

HQL also has features for working with unstructured data in HDFS

The above figure shows the connection of Hive to Hadoop (HDFS + Map Reduce) and the internal structure of Hive.

The main components of Hive are:

  • Metastore: It stores all the metadata of Hive. It stores data of data stored in database, tables, columns, etc.
  • Driver: It includes compiler, optimizer and executor used to break down the Hive query language statements.
  • Query compiler: It compiles HiveQL into graph of map reduce tasks.
  • Execution engine: It executes the tasks produces by compiler.
  • Thrift server: It provides an interface to connect to other applications like MySQL, Oracle, Excel, etc. through JDBC/ODBC drivers.
  • Command line interface: It is also called Hive shell. It is used for working with data either interactively or batch data processing.
  • Web Interface: It is a visual structure on Hive used for interaction with data.

Data Storage in Hive

Hive has different forms of storage options and they include:

  1. Metastore: Metastore keeps track of all the metadata of database, tables, columns, datatypes etc. in Hive. It also keeps track of HDFS mapping.
  2. Tables: There can be 2 types of tables in Hive. First, normal tables like any other table in database. Second, external tables which are like normal tables except for the deletion part. HDFS mappings are used to create external tables which are pointers to table in HDFS. The difference between the two types of tables is that when the external table is deleted its data is not deleted. Its data is stored in the HDFS whereas in case of normal table the data also gets deleted on deleting the table.
    Partitions: Partition is slicing of tables that are stored in different subdirectory within a table’s directory. It enhances query performance especially in case of select statements with “WHERE” clause.
    Buckets: Buckets are hashed partitions and they speed up joins and sampling of data.

Hive vs. RDBMS (Relational database)

Hive and RDBMS are very similar but they have different applications and different schemas that they are based on.

  • Hive is built for OLAP that is real time reporting of data. Hive does not support inserting into an existing table or updating table data like RDBMS which is an important part of OLTP process. All data is either inserted in new table or overwritten in existing table.
  • Hive is based on read schema that means data is not checked when it is loaded so data loading is fast but reading is slower.

Hive Query Language (HQL)

Load is used for taking data from HDFS and moving it into Hive. Insert is used for moving data from one Hive table to another. Select is used for querying data. Explain gives insights into structure of data.

Hive Installation

Prerequisites for installing Hive:

  • Java 1.7
  • Hadoop 2.x

Steps to install:

  1. Download stable version of Hive from http://Hive.apache.org/
  2. Go to downloads and select the latest mirror. Download the latest tar ball apache-Hive-1.2.1-bin.tar.gz
  3. Unzip the tar ball using following command: tar -xzvf tar -xzvf apache-Hive-1.2.1-bin.tar.gz
  4. Set the environment variable HIVE_HOME to point to the installation directory:
    export HIVE_HOME = /user/local/Hive

Finally, add HIVE_HOME/bin to PATH:

export PATH = $PATH:HIVE_HOME/bin

To start Hive shell just type Hive after setting the path and Hive shell will fire up. To verify that Hive has started use command:

set –v

All the Hive properties will show up and look for mapred.job.tracker = hname : 1002 to verify that Hive has found the Hadoop cluster. Thus Hive is installed successfully and database can be created followed by tables and queries.

 

 

 

 

 

 

 

 

 

 

Advertisements

From → Data Science, General

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: