{"id":194,"date":"2017-09-28T12:23:46","date_gmt":"2017-09-28T12:23:46","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=194"},"modified":"2017-09-28T12:23:46","modified_gmt":"2017-09-28T12:23:46","slug":"hadoop-hdfs-data-block","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/","title":{"rendered":"HDFS Data Block – Learn the Internals of Big Data Hadoop"},"content":{"rendered":"

In this Big data Hadoop tutorial<\/strong>, we are going to provide you a detailed description of Hadoop HDFS data block. First of all, we will cover what is data block in Hadoop, what is their importance, why the size of HDFS data blocks is 128MB.<\/p>\n

We will also discuss the example of data blocks in hadoop and various advantages of HDFS in Hadoop.<\/p>\n

\"HDFS<\/a><\/p>\n

Introduction to HDFS Data Block<\/h3>\n

Hadoop HDFS<\/a><\/strong> split large files into small chunks known as Blocks<\/strong>. Block is the physical representation of data. It contains a minimum amount of data that can be read or write. HDFS stores each file as blocks. HDFS client doesn’t have any control on the block like block location, Namenode decides all such things.<\/p>\n

By default, HDFS block size is 128MB<\/strong> which you can change as per your requirement. All HDFS blocks are the same size except the last block, which can be either the same size or smaller.<\/p>\n

Hadoop framework break files into 128 MB blocks and then stores into the Hadoop file system. Apache Hadoop application is responsible for distributing the data block across multiple nodes.<\/p>\n

Example-<\/strong><\/p>\n

Suppose file size is 513MB, and we are using the default configuration of block size 128MB. Then, the Hadoop framework will create 5 blocks, first four blocks 128MB, but the last block will be of 1MB only.<\/p>\n

Hence from the example it clear that it is not necessary that in HDFS\u00a0each file stored should be an exact multiple of the configured block size 128mb, 256mb etc. Therefore final block for file uses only as much space as is needed.<\/p>\n

Why is HDFS Block size 128 MB?<\/h3>\n

HDFS store terabytes and petabytes of data. If HDFS Block size is 4kb like\u00a0Linux file system, then we will have too many data blocks in Hadoop HDFS, hence too much of metadata.<\/p>\n

So, maintaining and managing this huge number of blocks and metadata will create huge overhead and traffic which is something which we don\u2019t want.<\/p>\n

Block size can\u2019t be so large that the system is waiting a very long time for one last unit of data processing to finish its work.<\/p>\n

Advantages of HDFS<\/h3>\n

After learning what is HDFS Data Block, let’s now discuss the advantages of Hadoop HDFS.<\/p>\n

1. Ability to store very large files<\/h4>\n

Hadoop HDFS store very large files which are even larger than the size of a single disk as Hadoop framework break file into blocks and distribute across various nodes.<\/p>\n

2. Fault tolerance and High Availability of HDFS<\/h4>\n

Hadoop framework can easily replicate Blocks between the datanodes. Thus provide\u00a0fault tolerance\u00a0and high availability<\/strong> HDFS.<\/p>\n

3. Simplicity of storage management<\/h4>\n

As HDFS has fixed block size (128MB), so it is very easy to calculate the number of blocks that can be stored on the disk.<\/p>\n

4. Simple Storage mechanism for datanodes<\/h4>\n

Block in HDFS simplifies the storage of the Datanodes<\/strong>. Namenode<\/strong> maintains metadata of all the blocks. HDFS Datanode does not need to concern about the block metadata like file permissions etc.<\/p>\n

Conclusion<\/h3>\n

Hence, HDFS data block is the smallest unit of data in a filesystem. The default size of the HDFS Block is 128MB which you can configure as per requirement. HDFS Blocks are easy to replicate between the datanodes. Hence, provide fault tolerance and high availability of HDFS.<\/p>\n

For any query or suggestion related to Hadoop HDFS data blocks, do let us know by leaving a comment in a section given below.<\/p>\n","protected":false},"excerpt":{"rendered":"

In this Big data Hadoop tutorial, we are going to provide you a detailed description of Hadoop HDFS data block. First of all, we will cover what is data block in Hadoop, what is...<\/p>\n","protected":false},"author":1,"featured_media":73154,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[564],"tags":[457,541,556,566,543,557,567],"class_list":["post-194","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hdfs","tag-big-data","tag-hadoop","tag-hadoop-hdfs","tag-hadoop-hdfs-data-blocks","tag-hadoop-tutorial","tag-hdfs","tag-hdfs-data-blocks"],"yoast_head":"\nHDFS Data Block - Learn the Internals of Big Data Hadoop - TechVidvan<\/title>\n<meta name=\"description\" content=\"Hadoop HDFS data block introduction covers the reason to set block size 128MB, need of data block, advantages of Hadoop HDFS, Hadoop HDFS data block example\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"HDFS Data Block - Learn the Internals of Big Data Hadoop - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Hadoop HDFS data block introduction covers the reason to set block size 128MB, need of data block, advantages of Hadoop HDFS, Hadoop HDFS data block example\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-28T12:23:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Data-Block-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"HDFS Data Block - Learn the Internals of Big Data Hadoop - TechVidvan","description":"Hadoop HDFS data block introduction covers the reason to set block size 128MB, need of data block, advantages of Hadoop HDFS, Hadoop HDFS data block example","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/","og_locale":"en_US","og_type":"article","og_title":"HDFS Data Block - Learn the Internals of Big Data Hadoop - TechVidvan","og_description":"Hadoop HDFS data block introduction covers the reason to set block size 128MB, need of data block, advantages of Hadoop HDFS, Hadoop HDFS data block example","og_url":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2017-09-28T12:23:46+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Data-Block-01.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"HDFS Data Block – Learn the Internals of Big Data Hadoop","datePublished":"2017-09-28T12:23:46+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/"},"wordCount":583,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Data-Block-01.jpg","keywords":["big data","hadoop","Hadoop HDFS","Hadoop HDFS Data blocks","hadoop tutorial","hdfs","HDFS Data blocks"],"articleSection":["HDFS Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/","url":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/","name":"HDFS Data Block - Learn the Internals of Big Data Hadoop - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Data-Block-01.jpg","datePublished":"2017-09-28T12:23:46+00:00","description":"Hadoop HDFS data block introduction covers the reason to set block size 128MB, need of data block, advantages of Hadoop HDFS, Hadoop HDFS data block example","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Data-Block-01.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HDFS-Data-Block-01.jpg","width":1200,"height":628,"caption":"HDFS Data Block"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-hdfs-data-block\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"HDFS Data Block – Learn the Internals of Big Data Hadoop"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/194","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=194"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/194\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73154"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=194"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=194"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=194"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}