{"id":254,"date":"2017-10-03T05:40:05","date_gmt":"2017-10-03T05:40:05","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=254"},"modified":"2017-10-03T05:40:05","modified_gmt":"2017-10-03T05:40:05","slug":"hadoop-counters-types-and-roles","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/","title":{"rendered":"Hadoop Counters & Types of Counters in MapReduce"},"content":{"rendered":"

In our previous Hadoop<\/strong> blog we have provided you a detailed description of Hadoop InputFormat<\/strong> and OutputFormat<\/strong>. Now we are going to cover Hadoop\u00a0Counters in detail. In this Hadoop tutorial we will discuss what is MapReduce Counters, What are their roles.<\/p>\n

At last we will also cover the types of Counters in Hadoop MapReduce. Such as\u00a0MapReduce Task Counter,\u00a0File System Counters,\u00a0FileInputFormat Counters,\u00a0<\/strong>FileOutputFormat counters,\u00a0Job Counters in MapReduce,\u00a0Dynamic Counters in Hadoop.<\/p>\n

\"hadoop<\/a><\/p>\n

Hadoop MapReduce<\/h3>\n

Before we start with Hadoop Counters, let us first learn what is Hadoop MapReduce?<\/p>\n

MapReduce<\/strong> is the data processing layer of Hadoop. It processes large structured and unstructured data stored in HDFS.<\/strong> MapReduce also processes a huge amount of data in parallel. It does this by dividing the job (submitted job) into a set of independent tasks (sub-job). In Hadoop, MapReduce works by breaking the processing into phases: Map<\/strong> and Reduce<\/strong>.<\/p>\n

    \n
  • Map Phase-<\/strong> It is the first phase of the data process. In this phase, we specify all the complex logic\/business rules\/costly code.<\/li>\n
  • Reduce Phase-<\/strong> It is the second phase of processing. In this phase, we specify light-weight processing like aggregation\/summation.<\/li>\n<\/ul>\n

    What is Hadoop Counters?<\/h3>\n

    Counters<\/strong> in Hadoop are a useful channel for gathering statistics about the MapReduce job. Like for quality control or for application-level. Counters are also useful for problem diagnosis.<\/p>\n

    A Counter represents Apache Hadoop global counters, defined either by the MapReduce framework. Each counter in MapReduce is named by an \u201cEnum\u201d. It also has a long for the value.<\/p>\n

    Hadoop Counters validate that:<\/p>\n

      \n
    • It reads and written correct number of bytes.<\/li>\n
    • It has launched and successfully run correct number of tasks or not.<\/li>\n
    • Counters also validate that the amount of CPU and memory consumed is appropriate for our job and cluster nodes or not.<\/li>\n<\/ul>\n

      Types of Counters in MapReduce<\/h3>\n

      2 types of MapReduce counters are:<\/p>\n

        \n
      • Built-in Counters<\/li>\n
      • User-Defined Counters\/Custom counters<\/li>\n<\/ul>\n

        1. Built-in Counters in Hadoop MapReduce<\/h4>\n

        Apache Hadoop<\/strong> maintains some built-in counters for every job. These counters report various metrics. There are counters for the number of bytes and records. Which allow us to confirm that the expected amount of input is consumed and the expected amount of output is produced.<\/p>\n

        Hadoop Counters are also divided into groups. There are several groups of the built-in counters. Each group also either contains task counters or contain job counter.<\/p>\n

        Several groups of the built-in counters in Hadoop are as follows:<\/p>\n

        a) MapReduce Task Counter<\/h5>\n

        Task counter collects specific information about tasks during its execution time. Which include the number of records read and written.<\/p>\n

        For example MAP_INPUT_RECORDS counter is the Task Counter. It also counts the input records read by each map task.<\/p>\n

        b) File System Counters<\/h5>\n

        This Counter gathers information like a number of bytes read and written by the file system. The name and description of the file system counters are as follows:<\/p>\n

          \n
        • FileSystem bytes read<\/strong>\u2013 The number of bytes read by the filesystem.<\/li>\n
        • FileSystem bytes written<\/strong>\u2013 The number of bytes written to the filesystem.<\/li>\n<\/ul>\n
          c) FileInputFormat Counters<\/h5>\n

          These Counters also gather information of a number of bytes read by map tasks via FileInputFormat.<\/p>\n

          d) FileOutputFormat counters<\/h5>\n

          These counters also gather information of a number of bytes written by map tasks (for map-only jobs) or reduce tasks via FileOutputFormat.<\/p>\n

          e) Job Counters in MapReduce<\/h5>\n

          Job counter measures\u00a0the job-level statistics. It does not measure values that change while a task is running.<\/p>\n

          For example TOTAL_LAUNCHED_MAPS, count the number of map tasks that were launched over the course of a job. Application master also measures the Job counters.<\/p>\n

          So they don\u2019t need to be sent across the network, unlike all other counters, including user-defined ones.<\/p>\n

          2. User-Defined Counters or Custom Counters in Hadoop MapReduce<\/h4>\n

          In addition to built-in counters, Hadoop MapReduce permits user code to define a set of counters. Then it increment them as desired in the\u00a0mapper<\/strong>\u00a0or\u00a0reducer<\/strong>. Like in Java to define counters it uses, \u2018enum<\/strong>\u2019 .<\/p>\n

          A job may define an arbitrary number of \u2018enums\u2019. Each with an arbitrary number of fields. The name of the enum is the group name. The enum\u2019s fields are the counter names.<\/p>\n

          a) Dynamic Counters in Hadoop<\/h5>\n

          Java enum\u2019s fields are defined at compile time. So we cannot create new counters at run time using enums. So, we use dynamic counters to create new counters at run time. But dynamic counter is not defined at compile time.<\/p>\n

          Conclusion<\/h3>\n

          Hence, Counters check whether it has read and written correct number of bytes. Counter also measures the progress or the number of operations that occur within MapReduce job.<\/p>\n

          Hadoop also maintains built-in counters and user-defined counters to measure the progress that occurs within MapReduce job.<\/p>\n

          Hope this blog helped you, If you have any query related to Hadoop Counter so, leave a comment in a section below.<\/p>\n","protected":false},"excerpt":{"rendered":"

          In our previous Hadoop blog we have provided you a detailed description of Hadoop InputFormat and OutputFormat. Now we are going to cover Hadoop\u00a0Counters in detail. In this Hadoop tutorial we will discuss what...<\/p>\n","protected":false},"author":1,"featured_media":73139,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[570],"tags":[538,457,539,541,574,575,543],"class_list":["post-254","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-mapreduce","tag-apache-hadoop","tag-big-data","tag-big-data-hadoop","tag-hadoop","tag-hadoop-counters","tag-hadoop-mapreduce","tag-hadoop-tutorial"],"yoast_head":"\nHadoop Counters & Types of Counters in MapReduce - TechVidvan<\/title>\n<meta name=\"description\" content=\"Apache Hadoop Counters cover What is MapReduce,MapReduce Counters Introduction,Types of Counters in MapReduce-Built-in Counters,Custom counters in Hadoop MR\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Hadoop Counters & Types of Counters in MapReduce - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Apache Hadoop Counters cover What is MapReduce,MapReduce Counters Introduction,Types of Counters in MapReduce-Built-in Counters,Custom counters in Hadoop MR\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-10-03T05:40:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/hadoop-counters-type.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"802\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Hadoop Counters & Types of Counters in MapReduce - TechVidvan","description":"Apache Hadoop Counters cover What is MapReduce,MapReduce Counters Introduction,Types of Counters in MapReduce-Built-in Counters,Custom counters in Hadoop MR","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/","og_locale":"en_US","og_type":"article","og_title":"Hadoop Counters & Types of Counters in MapReduce - TechVidvan","og_description":"Apache Hadoop Counters cover What is MapReduce,MapReduce Counters Introduction,Types of Counters in MapReduce-Built-in Counters,Custom counters in Hadoop MR","og_url":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2017-10-03T05:40:05+00:00","og_image":[{"width":802,"height":420,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/hadoop-counters-type.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Hadoop Counters & Types of Counters in MapReduce","datePublished":"2017-10-03T05:40:05+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/"},"wordCount":811,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/hadoop-counters-type.jpg","keywords":["apache hadoop","big data","big data hadoop","hadoop","Hadoop Counters","hadoop mapreduce","hadoop tutorial"],"articleSection":["MapReduce Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/","url":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/","name":"Hadoop Counters & Types of Counters in MapReduce - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/hadoop-counters-type.jpg","datePublished":"2017-10-03T05:40:05+00:00","description":"Apache Hadoop Counters cover What is MapReduce,MapReduce Counters Introduction,Types of Counters in MapReduce-Built-in Counters,Custom counters in Hadoop MR","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/hadoop-counters-type.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/hadoop-counters-type.jpg","width":802,"height":420,"caption":"hadoop counters types"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-counters-types-and-roles\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Hadoop Counters & Types of Counters in MapReduce"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/254","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=254"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/254\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73139"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=254"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=254"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=254"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}