{"id":315,"date":"2017-10-04T05:10:50","date_gmt":"2017-10-04T05:10:50","guid":{"rendered":"http:\/\/techvidvan.com\/tutorials\/?p=315"},"modified":"2017-10-04T05:10:50","modified_gmt":"2017-10-04T05:10:50","slug":"hadoop-partitioner-introduction","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/hadoop-partitioner-introduction\/","title":{"rendered":"Hadoop Partitioner – Learn the Basics of MapReduce Partitioner"},"content":{"rendered":"

The main goal of this Hadoop Tutorial<\/strong> is to provide you a detailed description of each component\u00a0that is used in Hadoop working. In this tutorial, we are going to cover the Partitioner in Hadoop.<\/p>\n

What is Hadoop Partitioner, what is the need of Partitioner in Hadoop, What is the default Partitioner in MapReduce, How many MapReduce Partitioner are used in Hadoop?<\/p>\n

We will answer all these questions in this MapReduce tutorial.<\/p>\n

\"hadoop<\/a><\/p>\n

What is Hadoop Partitioner?<\/h3>\n

Partitioner in MapReduce job execution\u00a0controls the partitioning of the keys of the intermediate map-outputs. With the help of hash function, key (or a subset of the key) derives the partition. The total number of partitions is equal to the number of reduce tasks.<\/p>\n

On the basis of key value<\/strong>, framework partitions, each mapper<\/strong> output. Records as having the same key value go into the same partition (within each mapper). Then each partition is sent to a reducer<\/strong>.<\/p>\n

Partition class decides which partition a given (key, value) pair will go. Partition phase in MapReduce data flow takes place after map phase and before reduce phase.<\/p>\n

Need of MapReduce Partitioner in Hadoop<\/h3>\n

In MapReduce job execution, it\u00a0takes an input data set and produces the list of key value pair. These key-value pair is the result of map phase. In which input data are split and each task processes the split and each map, output the list of key value pairs.<\/p>\n

Then, framework sends the map output to reduce task. Reduce processes the user-defined reduce function on map outputs. Before reduce phase, partitioning of the map output take place on the basis of the key.<\/p>\n

Hadoop Partitioning specifies that all the values for each key are grouped together. It also makes sure that all the values of a single key go to the same reducer. This allows even distribution of the map output over the reducer.<\/p>\n

Partitioner in a MapReduce job redirects the mapper output to the reducer by determining which reducer handles the particular key.<\/p>\n

Hadoop Default Partitioner<\/h3>\n

Hash Partitioner<\/strong> is the default Partitioner. It computes a hash value for the key. It also assigns the partition based on this result.<\/p>\n

How many Partitioner in Hadoop?<\/h3>\n

The total number of Partitioner depends on the number of reducers. Hadoop Partitioner divides the data according to the number of reducers. It is set by JobConf.setNumReduceTasks()<\/strong> method.<\/p>\n

Thus the single reducer processes the data from single partitioner. The important thing to notice is that the framework creates partitioner only when there are many reducers.<\/p>\n

Poor Partitioning in Hadoop MapReduce<\/h3>\n

If in data input in MapReduce job one key appears more than any other key. In such case, to send data to the partition we use two mechanisms which are as follows:<\/p>\n