{"id":79622,"date":"2020-08-25T09:00:47","date_gmt":"2020-08-25T03:30:47","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=79622"},"modified":"2020-08-25T09:00:47","modified_gmt":"2020-08-25T03:30:47","slug":"apache-sqoop-merge","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/","title":{"rendered":"Sqoop Merge Tool to Combine Datasets"},"content":{"rendered":"

Sqoop Merge tool is for combining the two datasets. There are many other insights available for the Sqoop Merge. The article will explain what Sqoop Merge is, its syntax, arguments, and much more.<\/p>\n

So, let\u2019s start.<\/p>\n

\"Sqoop<\/a><\/p>\n

What is Sqoop Merge?<\/h3>\n

Sqoop Merge is a tool that allows us to combine two datasets. The entries of one dataset override the entries of the older dataset.<\/p>\n

It is useful for efficiently transferring the vast volume of data between Hadoop and structured data stores like relational databases. After performing the merge operation, we can import the data back into the Apache Hive or HBase.<\/p>\n

In short, the Sqoop merge tool “flatten” the two datasets into one, by taking the newest available records for each primary key.<\/p>\n

Sqoop Merge Syntax<\/h3>\n
$ sqoop merge (generic-args) (merge-args)\n$ sqoop-merge (generic-args) (merge-args)\n<\/pre>\n
The Hadoop generic arguments should be passed before any merge arguments. However, we can enter job arguments in any order with respect to each other.<\/p>\n
Merge Arguments and How Sqoop Merge works?<\/h3>\n
The merge arguments are:<\/p>\n\n\n\n\n\n\n\n\n\n
Argument<\/b><\/td>\nDescription<\/b><\/td>\n<\/tr>\n
–class-name <class><\/span><\/td>\nIt Specifies the name of the record-specific class to be used during the merge job.<\/span><\/td>\n<\/tr>\n
–jar-file <file><\/span><\/td>\nThis specifies the name of the jar from which the record class is to be loaded.<\/span><\/td>\n<\/tr>\n
–merge-key <col><\/span><\/td>\nIt specifies the name of the column to be used as the merge key.<\/span><\/td>\n<\/tr>\n
–new-data <path><\/span><\/td>\nIt will specify the path of the\u00a0 newer dataset.<\/span><\/td>\n<\/tr>\n
–onto <path><\/span><\/td>\nIt will specify the path of the older dataset.<\/span><\/td>\n<\/tr>\n
–target-dir <path><\/span><\/td>\nIt will specify the target path for the output of the merge job.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    \n
  • The Sqoop merge tool runs the MapReduce job, which takes two directories as an input. One is the newer dataset, and the other is the older dataset. These two directories are specified with the –new-data and –onto respectively.<\/li>\n
  • The output generated by this MapReduce job is placed in the directory in HDFS, which is specified by the –target-dir.<\/li>\n
  • While merging the datasets, it is assumed that there is a unique primary key value in each record.<\/li>\n
  • We specify the primary key column by –merge-key argument. More than one row in the same dataset must not have the same primary key. Otherwise, data may be lost.<\/li>\n
  • For parsing the dataset and extracting the key column, the auto-generated class from the previous import must be used.<\/li>\n
  • We can specify the class name and the jar file with the arguments –class-name and –jar-file. If it is not available, then we can recreate this class by using the Sqoop Codegen tool.<\/li>\n
  • The Sqoop merge tool typically runs after the incremental import with the date-last-modified mode, that is, (sqoop import \u2014incremental lastmodified \u2026).<\/li>\n<\/ul>\n
    Example Invocations<\/h3>\n
    Suppose we had performed the two incremental imports where some older data is in the HDFS directory named older, and the newer data is also in the HDFS directory named newer. Then we can merge this as:<\/p>\n
    $ sqoop merge --new-data newer --onto older --target-dir merged \\\n    --jar-file datatypes.jar --class-name Foo --merge-key id\n<\/pre>\n
    This command runs the MapReduce job where the value in the id column of each row is used for joining rows. The rows in a newer dataset are used in preference to the rows in an older dataset.<\/p>\n
    Summary<\/h3>\n
    I hope after reading this article, you have clearly understood the entire concept of the Sqoop Merge tool. This tool is useful for combining the two datasets.<\/p>\n
    The entries of one dataset overrides the entries of the older dataset. The article had explained the Sqoop Merge tool syntax, arguments, as well as its working.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Sqoop Merge tool is for combining the two datasets. There are many other insights available for the Sqoop Merge. The article will explain what Sqoop Merge is, its syntax, arguments, and much more. So,...<\/p>\n","protected":false},"author":1,"featured_media":79669,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3163],"tags":[3182,3183,3184],"class_list":["post-79622","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sqoop","tag-example-of-sqoop-merge","tag-purpose-of-sqoop-merge","tag-what-is-sqoop-merge"],"yoast_head":"\nSqoop Merge Tool to Combine Datasets - TechVidvan<\/title>\n<meta name=\"description\" content=\"Sqoop Merge is a tool that allows us to combine two datasets. Learn Sqoop Merge tool syntax, arguments, as well as its working with Examples,\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Sqoop Merge Tool to Combine Datasets - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Sqoop Merge is a tool that allows us to combine two datasets. Learn Sqoop Merge tool syntax, arguments, as well as its working with Examples,\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-25T03:30:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2020\/08\/Sqoop-Merge-tv.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Sqoop Merge Tool to Combine Datasets - TechVidvan","description":"Sqoop Merge is a tool that allows us to combine two datasets. Learn Sqoop Merge tool syntax, arguments, as well as its working with Examples,","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/","og_locale":"en_US","og_type":"article","og_title":"Sqoop Merge Tool to Combine Datasets - TechVidvan","og_description":"Sqoop Merge is a tool that allows us to combine two datasets. Learn Sqoop Merge tool syntax, arguments, as well as its working with Examples,","og_url":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2020-08-25T03:30:47+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2020\/08\/Sqoop-Merge-tv.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Sqoop Merge Tool to Combine Datasets","datePublished":"2020-08-25T03:30:47+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/"},"wordCount":580,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2020\/08\/Sqoop-Merge-tv.jpg","keywords":["example of sqoop merge","Purpose of Sqoop Merge","what is Sqoop Merge"],"articleSection":["Sqoop Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/","url":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/","name":"Sqoop Merge Tool to Combine Datasets - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2020\/08\/Sqoop-Merge-tv.jpg","datePublished":"2020-08-25T03:30:47+00:00","description":"Sqoop Merge is a tool that allows us to combine two datasets. Learn Sqoop Merge tool syntax, arguments, as well as its working with Examples,","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2020\/08\/Sqoop-Merge-tv.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2020\/08\/Sqoop-Merge-tv.jpg","width":1200,"height":628,"caption":"Sqoop Merge"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/apache-sqoop-merge\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Sqoop Merge Tool to Combine Datasets"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/79622","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=79622"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/79622\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/79669"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=79622"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=79622"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=79622"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}