{"id":1992,"date":"2017-09-29T08:52:45","date_gmt":"2017-09-29T08:52:45","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=220"},"modified":"2017-09-29T08:52:45","modified_gmt":"2017-09-29T08:52:45","slug":"hadoop-reducer","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/","title":{"rendered":"What is Hadoop Reducer Class in MapReduce?"},"content":{"rendered":"

Till now we have covered the Hadoop introduction<\/strong> and Hadoop HDFS<\/strong> in detail. In this tutorial, we will provide you a detailed description of Hadoop Reducer.<\/p>\n

Here will discuss what is Reducer in MapReduce, how Reducer works in Hadoop MapReduce, different phases of Hadoop Reducer, how\u00a0can we change the number of Reducer\u00a0in Hadoop MapReduce.<\/p>\n

\"Hadoop<\/a><\/p>\n

What is Hadoop Reducer?<\/h3>\n

Reducer<\/strong> in Hadoop MapReduce reduces a set of intermediate values which share a key to\u00a0a smaller set of values.<\/p>\n

In MapReduce job execution flow, Reducer takes a set of an intermediate key-value pair<\/strong> produced by the mapper<\/strong><\/a> as the input. Then, Reducer aggregate, filter and combine key-value pairs and this requires a wide range of processing.<\/p>\n

One-one mapping takes place between keys and reducers in MapReduce job execution. They run in parallel since they are independent of one another. The user decides the number of reducers in MapReduce.<\/p>\n

Phases of Hadoop Reducer<\/strong><\/h3>\n

Three phases of Reducer are as follows:<\/p>\n

1. Shuffle Phase<\/h4>\n

This is the phase in which sorted output from the mapper is the input to the reducer. The framework with the help of HTTP fetches the relevant partition of the output of all the mappers in this phase.Sort phase<\/p>\n

2. Sort Phase<\/h4>\n

This is the phase in which the input from different mappers is again sorted based on the similar keys in different Mappers.<\/p>\n

Both Shuffle and Sort occur concurrently.<\/p>\n

3. Reduce Phase<\/h4>\n

This phase occurs after shuffle and sort. Reduce task aggregates the key-value pairs. With the OutputCollector.collect()<\/strong> <\/em>property, the output of the reduce task is written to the FileSystem. Reducer output is not sorted.<\/p>\n

Number of Reducers in Hadoop MapReduce<\/h3>\n

User set the number of reducers with the help of Job.setNumreduceTasks(int)<\/strong> property. Thus the right number of reducers by the formula:<\/p>\n

\u00a00.95 or 1.75 multiplied by (<no. of nodes> * <no. of the maximum container per node>)<\/strong><\/p>\n

So, with 0.95, all reducers immediately launch. Then, start transferring map outputs as the maps finish.<\/p>\n

Faster node finishes the first round of reducers with 1.75. Then it launches the second wave of reducer which does much better job of load balancing.<\/p>\n

With the increase of the number of reducers:<\/strong><\/p>\n

    \n
  • Framework overhead increases.<\/li>\n
  • Load balancing increases.<\/li>\n
  • Cost of failures decreases.<\/li>\n<\/ul>\n

    Conclusion<\/h3>\n

    Hence, Reducer takes mappers output as input. Then, process the key-value pairs and produces the output. Reducer output is the final output. If you like this blog or you have any query related to Hadoop Reducer, so please share with us by leaving a comment.<\/p>\n

    Hope we will help you out.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Till now we have covered the Hadoop introduction and Hadoop HDFS in detail. In this tutorial, we will provide you a detailed description of Hadoop Reducer. Here will discuss what is Reducer in MapReduce,...<\/p>\n","protected":false},"author":1,"featured_media":73147,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[570],"tags":[538,457,583,575,569,463],"class_list":["post-1992","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-mapreduce","tag-apache-hadoop","tag-big-data","tag-big-data-hadoop-tutorial","tag-hadoop-mapreduce","tag-mapper-in-hadoop","tag-mapreduce"],"yoast_head":"\nWhat is Hadoop Reducer Class in MapReduce? - TechVidvan<\/title>\n<meta name=\"description\" content=\"Hadoop Reducer tutorial covers what is reducer in Hadoop mapreduce,phases of mapreduce reducer in Hadoop,shuffle Phase,sort phase,reducer phase in Hadoop MR\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is Hadoop Reducer Class in MapReduce? - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Hadoop Reducer tutorial covers what is reducer in Hadoop mapreduce,phases of mapreduce reducer in Hadoop,shuffle Phase,sort phase,reducer phase in Hadoop MR\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-29T08:52:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Hadoop-Reducer-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is Hadoop Reducer Class in MapReduce? - TechVidvan","description":"Hadoop Reducer tutorial covers what is reducer in Hadoop mapreduce,phases of mapreduce reducer in Hadoop,shuffle Phase,sort phase,reducer phase in Hadoop MR","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/","og_locale":"en_US","og_type":"article","og_title":"What is Hadoop Reducer Class in MapReduce? - TechVidvan","og_description":"Hadoop Reducer tutorial covers what is reducer in Hadoop mapreduce,phases of mapreduce reducer in Hadoop,shuffle Phase,sort phase,reducer phase in Hadoop MR","og_url":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2017-09-29T08:52:45+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Hadoop-Reducer-01.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"What is Hadoop Reducer Class in MapReduce?","datePublished":"2017-09-29T08:52:45+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/"},"wordCount":431,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Hadoop-Reducer-01.jpg","keywords":["apache hadoop","big data","Big data Hadoop tutorial","hadoop mapreduce","Mapper in Hadoop","MapReduce"],"articleSection":["MapReduce Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/","url":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/","name":"What is Hadoop Reducer Class in MapReduce? - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Hadoop-Reducer-01.jpg","datePublished":"2017-09-29T08:52:45+00:00","description":"Hadoop Reducer tutorial covers what is reducer in Hadoop mapreduce,phases of mapreduce reducer in Hadoop,shuffle Phase,sort phase,reducer phase in Hadoop MR","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Hadoop-Reducer-01.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Hadoop-Reducer-01.jpg","width":1200,"height":628,"caption":"Hadoop Reducer Class in MapReduce"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-reducer\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"What is Hadoop Reducer Class in MapReduce?"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/1992","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=1992"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/1992\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73147"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=1992"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=1992"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=1992"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}