{"id":2002,"date":"2017-10-06T04:41:29","date_gmt":"2017-10-06T04:41:29","guid":{"rendered":"http:\/\/techvidvan.com\/tutorials\/?p=392"},"modified":"2017-10-06T04:41:29","modified_gmt":"2017-10-06T04:41:29","slug":"hdfs-federation-introduction-architecture","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/","title":{"rendered":"Introduction to HDFS Federation &amp; Architecture"},"content":{"rendered":"<p>In this<strong> Hadoop tutorial<\/strong>, we are going to provide you a complete introduction of HDFS Federation. In this tutorial we will discuss HDFS Architecture, Limitations of the current architecture of HDFS.<\/p>\n<p>Then after we will cover the HDFS Federation architecture in detail along with their advantages in Hadoop framework.<\/p>\n<h3>What is HDFS Federation?<\/h3>\n<p><strong>Federation<\/strong>\u00a0enhances an existing <strong>Hadoop HDFS<\/strong> architecture. Prior HDFS architecture allows single namespace for the entire cluster. In that architecture, single NameNode manages namespace.<\/p>\n<p>If NameNode fails, then whole cluster will be out of service. And the cluster will be unavailable until the NameNode restarts or brought on a separate machine.<\/p>\n<p>HDFS Federation was introduced to overcome this limitation. It overcomes this by adding support for many NameNode\/Namespaces to HDFS.<\/p>\n<h3>Current HDFS Architecture<\/h3>\n<p>HDFS has two main layers given below:<\/p>\n<p><a href=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2019\/11\/hdfs-layers.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-73163\" src=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2019\/11\/hdfs-layers.jpg\" alt=\"hdfs layers\" width=\"802\" height=\"420\" \/><\/a><\/p>\n<p><strong>a) Namespace<\/strong><strong>\u2013<\/strong> This layer manages files, directories, and\u00a0<a href=\"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/\"><strong>blocks<\/strong><\/a>. This layer supports basic file system operation such as creation, deletion of files.<\/p>\n<p><strong>b) Block Storage<\/strong><strong>\u2013<\/strong> It has two parts-<\/p>\n<ul>\n<li><strong> Block management &#8211;\u00a0<\/strong>It supports block related operation such as creation, deletion of the blocks. It manages data nodes in the cluster and takes care of replication management.<\/li>\n<\/ul>\n<ul>\n<li><strong> Physical storage &#8211;\u00a0<\/strong>This stores the blocks on the local file system and provides access to read or write operation. Follow this link to\u00a0learn HDFS data read and write operation.<\/li>\n<\/ul>\n<p>This current HDFS works fine for smaller setups. But, for large organizations where we need to take care of the huge amount of data has some limitation. Hadoop federation handles those limitations.<\/p>\n<h3>Limitation of current HDFS Architecture<\/h3>\n<p>Limitation of current HDFS Architecture is given below:<\/p>\n<h4>1. Tightly coupled block storage and Namespace<\/h4>\n<p><em>Namespace layer<\/em>\u00a0and<em>\u00a0storage layer<\/em>\u00a0are tightly coupled. It makes alternate implementation of namenode difficult. And it restricts other services to use block storage.<\/p>\n<h4>2. Namespace Scalability<\/h4>\n<p>The namespace is not scalable like datanode. Scaling in HDFS cluster is horizontally by adding datanodes. But we can\u2019t add more namespace to an existing cluster. We can vertically scale namespace on a single namenode.<\/p>\n<h4>3. Performance<\/h4>\n<p>Hadoop entire performance depends on the throughput of the namenode. An operation of current file system depends on the throughput of a single namenode. NameNode at present supports 60,000 concurrent tasks.<\/p>\n<p>Upcoming\u00a0<strong>MapReduce<\/strong>\u00a0will have support for more than 1,00,000 concurrent tasks. And this will need more namenode.<\/p>\n<h4>4. Isolation<\/h4>\n<p>There is no separation of the namespace. So there is no isolation among tenant organization that is using the cluster.<\/p>\n<h3>HDFS<strong> Federation Architecture<\/strong><\/h3>\n<p>Federation uses many independent Namenode\/namespaces to scale the name service horizontally. In HDFS Federation Architecture, at the bottom, datanodes are present. And datanodes are used as a common storage for blocks by all the namenodes.<\/p>\n<p>Each datanodes registers with all the namenodes in the cluster. These datanodes send periodic heartbeats, block, report and handle command from the namenodes.<\/p>\n<p><a href=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2019\/11\/hdfs-federation-architecture.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-73162\" src=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2019\/11\/hdfs-federation-architecture.jpg\" alt=\"hdfs federation architecture\" width=\"802\" height=\"420\" \/><\/a><\/p>\n<p>Many namenodes (NN1, NN2\u2026, NNn) manages many namespaces (NS1, NS2\u2026, NSn) respectively. Each namespace has its own block pool (NS1 Has pool 1and so on). Block from pool 1 is stored on datanode 1 and so on.<\/p>\n<h4>1. Block pool<\/h4>\n<p>Set of blocks is\u00a0<strong>Block pool<\/strong>\u00a0that belongs to a single namespace. There is a collection of pools in HDFS federation architecture. And each block is managed from the other.<\/p>\n<p>This allows a namespace to create\u00a0<em>a Block ID<\/em>\u00a0for new blocks without coordination with another namespace. All Datanodes store data blocks present in all block pools.<\/p>\n<h4>2. Namespace volume<\/h4>\n<p>Namespace along with its block pool is\u00a0<strong>Namespace volume<\/strong>. Many namespace volumes are there in HDFS federation. Hence, each namespace volume works independently. When we delete namenode or namespace, then corresponding block pool present on the datanodes will also be deleted.<\/p>\n<h3>Benefits of HDFS Federation<\/h3>\n<p>HDFS Federation overcomes the limitations of prior HDFS architecture. Hence it provides:<\/p>\n<ul>\n<li><strong>Isolation \u2013\u00a0<\/strong>There is no isolation in single namenode in a multi-user environment. In HDFS federation different categories of application and users can be isolated to different namespaces by using many namenodes.<\/li>\n<li><strong>Namespace Scalability \u2013\u00a0<\/strong>In federation many namenodes horizontally scales up in the filesystem namespace.<\/li>\n<li><strong>Performance \u2013\u00a0<\/strong>We can improve Read\/write operation throughput by adding more namenodes.<\/li>\n<\/ul>\n<h3>Conclusion<\/h3>\n<p>In conclusion to HDFS Federation, we can say that it overcomes the limitation of single node HDFS architecture. In prior HDFS architecture for an entire cluster allows only single namespace. While Federation uses many independent Namenode\/namespaces to scale the name service horizontally.<\/p>\n<p>It also separates the\u00a0<strong>namespace layer<\/strong>\u00a0and the\u00a0<strong>storage<\/strong>\u00a0layer.\u00a0Hence provides Isolation, Scalability and simple design.<\/p>\n<p>If you have any query or suggestion related to the Federation in Hadoop HDFS, so let us know by leaving a comment.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this Hadoop tutorial, we are going to provide you a complete introduction of HDFS Federation. In this tutorial we will discuss HDFS Architecture, Limitations of the current architecture of HDFS. Then after we&#46;&#46;&#46;<\/p>\n","protected":false},"author":1,"featured_media":73161,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[564],"tags":[630,457,631,541,558,632],"class_list":["post-2002","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hdfs","tag-apache-hadoop-tutorial","tag-big-data","tag-federation-in-hadoop-hdfs","tag-hadoop","tag-hdfs-architecture","tag-hdfs-federation"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Introduction to HDFS Federation &amp; Architecture - TechVidvan<\/title>\n<meta name=\"description\" content=\"What is HDFS Federation,HDFS Architecture,Drawbacks of HDFS Architecture,HDFS Federation architecture,block pool, Advantages of HDFS Federation architecture\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction to HDFS Federation &amp; Architecture - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"What is HDFS Federation,HDFS Architecture,Drawbacks of HDFS Architecture,HDFS Federation architecture,block pool, Advantages of HDFS Federation architecture\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-10-06T04:41:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Federation-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introduction to HDFS Federation &amp; Architecture - TechVidvan","description":"What is HDFS Federation,HDFS Architecture,Drawbacks of HDFS Architecture,HDFS Federation architecture,block pool, Advantages of HDFS Federation architecture","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/","og_locale":"en_US","og_type":"article","og_title":"Introduction to HDFS Federation &amp; Architecture - TechVidvan","og_description":"What is HDFS Federation,HDFS Architecture,Drawbacks of HDFS Architecture,HDFS Federation architecture,block pool, Advantages of HDFS Federation architecture","og_url":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2017-10-06T04:41:29+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Federation-01.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Introduction to HDFS Federation &amp; Architecture","datePublished":"2017-10-06T04:41:29+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/"},"wordCount":763,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Federation-01.jpg","keywords":["Apache Hadoop Tutorial","big data","Federation in Hadoop HDFS","hadoop","hdfs architecture","HDFS Federation"],"articleSection":["HDFS Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/","url":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/","name":"Introduction to HDFS Federation &amp; Architecture - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Federation-01.jpg","datePublished":"2017-10-06T04:41:29+00:00","description":"What is HDFS Federation,HDFS Architecture,Drawbacks of HDFS Architecture,HDFS Federation architecture,block pool, Advantages of HDFS Federation architecture","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Federation-01.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Federation-01.jpg","width":1200,"height":628,"caption":"HDFS Federation"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/hdfs-federation-introduction-architecture\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Introduction to HDFS Federation &amp; Architecture"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2002","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=2002"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2002\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73161"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=2002"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=2002"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=2002"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}