{"id":323,"date":"2017-10-04T07:21:57","date_gmt":"2017-10-04T07:21:57","guid":{"rendered":"http:\/\/techvidvan.com\/tutorials\/?p=323"},"modified":"2017-10-04T07:21:57","modified_gmt":"2017-10-04T07:21:57","slug":"hadoop-combiner-introduction-working-advantages","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/","title":{"rendered":"Hadoop Combiner Introduction, Working & Advantages"},"content":{"rendered":"

In this Hadoop tutorial<\/strong><\/a>, we will provide you a detailed description of Hadoop Combiner. First of all, we will see what is MapReduce Combiner, what is the key role of Combiner in MapReduce.<\/p>\n

Then we will discuss the example of MapReduce program with and without combiner in Hadoop. At last, we will also see some advantages and disadvantages of Combiner in MapReduce.<\/p>\n

\"Hadoop<\/a><\/p>\n

\u00a0What is Hadoop Combiner?<\/h3>\n

Combiner<\/strong> is also known as \u201cMini-Reducer<\/strong>\u201d that summarizes the Mapper<\/strong><\/a> output record with the same Key before passing to the Reducer<\/strong>.<\/a><\/p>\n

On a large dataset when we run MapReduce job. So Mapper generates large chunks of intermediate data. Then the framework passes this intermediate data on the Reducer for further processing.<\/p>\n

This leads to enormous network congestion. The Hadoop framework provides a function known as\u00a0Combiner\u00a0<\/strong>that plays a key role in reducing network congestion.<\/p>\n

The primary job of Combiner a \u201cMini-Reducer is to process the output data from the Mapper, before passing it to Reducer. \u00a0It runs after the mapper and before the Reducer. Its usage is optional.<\/p>\n

How does Combiner work in Hadoop?<\/h3>\n

Now let us learn how things change when we use the combiner in MapReduce?<\/p>\n

\"MapReduce<\/a><\/p>\n

As we see in above diagram no combiner is there. Input is split into two mappers. The framework generates 9 keys from the mappers.<\/p>\n

So, now we have (9 key\/value) intermediate data. Further mapper sends this key-value<\/strong><\/a> directly to the reducer. While sending data to the reducer, it consumes some network bandwidth. It takes more time to transfer data to reducer if the size of data is big.<\/p>\n

\"MapReduce<\/a><\/p>\n

Now from the above diagram, if we use a combiner in between mapper and reducer. Then combiner will shuffle 9 key\/value before sending it to the reducer. And then generates 4 key\/value pair as an output.<\/p>\n

Now, Reducer needs to process only 4 key\/value pair data which are generated from 2 combiners. Therefore reducer gets executed only 4 times to produce the final output. Thus, this increases the overall performance.<\/p>\n

Advantages of Combiner in MapReduce<\/h3>\n

Let’s now discuss the benefits of Hadoop Combiner in MapReduce.<\/p>\n

    \n
  • Use of combiner reduces the time taken for data transfer between mapper and reducer.<\/li>\n
  • Combiner improves the overall performance of the reducer.<\/li>\n
  • It decreases the amount of data that reducer has to process.<\/li>\n<\/ul>\n

    Disadvantages of Combiner in MapReduce<\/h3>\n

    There are also some disadvantages of Hadoop Combiner. Let’s now discuss the same.<\/p>\n

      \n
    • In the local filesystem, when Hadoop stores the key-value pairs and run the combiner later this will cause expensive disk IO.<\/li>\n
    • MapReduce jobs can\u2019t depend on the combiner execution as there is no guarantee in its execution.<\/li>\n<\/ul>\n

      Conclusion<\/h3>\n

      Hence, Hadoop Combiner plays a key role in reducing network congestion. It improves the overall performance of the reducer by summarizing the output of Mapper.<\/p>\n

      I Hope now you have a clear understanding of Hadoop Combiner. If still you have any query, so, please let us know be leaving a comment in a section below.<\/p>\n","protected":false},"excerpt":{"rendered":"

      In this Hadoop tutorial, we will provide you a detailed description of Hadoop Combiner. First of all, we will see what is MapReduce Combiner, what is the key role of Combiner in MapReduce. Then...<\/p>\n","protected":false},"author":1,"featured_media":73084,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[570],"tags":[457,583,605,541,606,543],"class_list":["post-323","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-mapreduce","tag-big-data","tag-big-data-hadoop-tutorial","tag-combiner-in-mapreduce","tag-hadoop","tag-hadoop-combiner","tag-hadoop-tutorial"],"yoast_head":"\nHadoop Combiner Introduction, Working & Advantages - TechVidvan<\/title>\n<meta name=\"description\" content=\"Learn what is Hadoop Combiner,Need of combiner in mapreduce,How MapReduce Combiner work in hadoop,Combiner benefits & limitations,Mapreduce Combiner Example\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Hadoop Combiner Introduction, Working & Advantages - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Learn what is Hadoop Combiner,Need of combiner in mapreduce,How MapReduce Combiner work in hadoop,Combiner benefits & limitations,Mapreduce Combiner Example\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-10-04T07:21:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Combiner-In-Hadoop-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Hadoop Combiner Introduction, Working & Advantages - TechVidvan","description":"Learn what is Hadoop Combiner,Need of combiner in mapreduce,How MapReduce Combiner work in hadoop,Combiner benefits & limitations,Mapreduce Combiner Example","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/","og_locale":"en_US","og_type":"article","og_title":"Hadoop Combiner Introduction, Working & Advantages - TechVidvan","og_description":"Learn what is Hadoop Combiner,Need of combiner in mapreduce,How MapReduce Combiner work in hadoop,Combiner benefits & limitations,Mapreduce Combiner Example","og_url":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2017-10-04T07:21:57+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Combiner-In-Hadoop-01.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Hadoop Combiner Introduction, Working & Advantages","datePublished":"2017-10-04T07:21:57+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/"},"wordCount":498,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Combiner-In-Hadoop-01.jpg","keywords":["big data","Big data Hadoop tutorial","Combiner in MapReduce","hadoop","Hadoop combiner","hadoop tutorial"],"articleSection":["MapReduce Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/","url":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/","name":"Hadoop Combiner Introduction, Working & Advantages - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Combiner-In-Hadoop-01.jpg","datePublished":"2017-10-04T07:21:57+00:00","description":"Learn what is Hadoop Combiner,Need of combiner in mapreduce,How MapReduce Combiner work in hadoop,Combiner benefits & limitations,Mapreduce Combiner Example","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Combiner-In-Hadoop-01.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Combiner-In-Hadoop-01.jpg","width":1200,"height":628,"caption":"Hadoop Combiner"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-combiner-introduction-working-advantages\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Hadoop Combiner Introduction, Working & Advantages"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/323","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=323"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/323\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73084"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=323"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=323"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=323"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}