{"id":2017,"date":"2018-01-09T13:09:53","date_gmt":"2018-01-09T13:09:53","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=695"},"modified":"2018-01-09T13:09:53","modified_gmt":"2018-01-09T13:09:53","slug":"hadoop-spark-compatibility","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/","title":{"rendered":"Hadoop Spark Compatibility: Hadoop+Spark better together"},"content":{"rendered":"

This tutorial is all about\u00a0Hadoop Spark Compatibility<\/em>. Hadoop and Spark together build a very powerful system to address all the Big Data requirements. Spark complements Hadoop with tons of power, you can handle all the diverse workloads, which was not possible with Hadoop’s MapReduce.<\/p>\n

Apache Spark is not developed to replace Hadoop rather it’s developed to complement Hadoop. Hadoop’s storage layer – HDFS is the most reliable storage system on the planet, on the other hand, its processing layer is quite limited to batch processing.<\/p>\n

Here Spark comes to rescue, using which we can handle: batch, real-time, streaming, graph, interactive, iterative requirements.<\/p>\n

 <\/p>\n

First of all, we discuss what is <\/span>Hadoop\u00a0and what is Apache Spark.<\/span><\/p>\n

\u00a0<\/span>Hadoop<\/strong> is an <\/span>open source software<\/em> framework for distributed storage & processing of huge amount of data sets, which also makes it possible to run applications on a system with thousands of nodes.<\/span><\/p>\n

\u00a0Apache Spark\u00a0is powerful\u00a0<\/span>cluster computing engine,\u00a0<\/em>which\u00a0is purposely designed for fast computation in Big Data world.<\/span><\/p>\n

Hadoop Spark Compatibility<\/strong>\u00a0is explaining all three modes to use Spark over Hadoop, such as Standalone, YARN, SIMR(Spark In MapReduce). To understand in detail we will learn by studying launching methods on all three modes.<\/p>\n

In closing, we will also cover the working of SIMR in Spark Hadoop compatibility.<\/p>\n

Spark and Hadoop working together<\/h3>\n

At first, we always get confused with the term replace<\/strong>, that spark replaced Hadoop. But to clarify spark does not replace Hadoop, it enhances the functionality of Hadoop. The day when spark comes in the picture it was the sketch to read and write data from and to HDFS.<\/p>\n

Even also with other storage systems like HBase and Amazon\u2019s S3. Although, Hadoop Users can enhance their processing capabilities by combining Hadoop with spark.<\/a> Below we will show Hadoop spark compatibility in detail.<\/p>\n

Hadoop – Spark Compatibility<\/h3>\n

It is easy as possible for every Hadoop user to take benefit of spark\u2019s capabilities. Hadoop spark compatibility does not affect either we run Hadoop 1.x or Hadoop 2.0 (YARN).<\/p>\n

No matter if we have privileges to configure the Hadoop cluster or not, there is a way for us to run Spark. In certain, there are three modes to deploy spark in a Hadoop cluster: Standalone, YARN,<\/strong> and SIMR<\/strong>.<\/p>\n

Let\u2019s understand all three ways one by one. Before that, the diagram below is also showing Hadoop spark compatibility.<\/p>\n

\"Spark<\/a><\/p>\n

Standalone :<\/h4>\n

When we use standalone deployment, we can statically allocate resources over the cluster. Also on a subset of machines in a Hadoop cluster. Even we can run spark side by side with Hadoop MR.<\/p>\n

Afterwards, the user can run arbitrary spark jobs on their HDFS data. Moreover, the simplicity of this deployment makes its choice for many Hadoop 1.x users.<\/p>\n

Hadoop Yarn :<\/h4>\n

This mode allows users to easily integrate spark in their Hadoop stack. It offers to take advantage of the full power of spark, as well as of other components running on top of spark.<\/p>\n

Hadoop users who are planning to deploy Hadoop yarn can simply run spark on yarn. Even users who are already deployed Hadoop yarn can also use spark easily. There is no pre-installation or administrative access required.<\/p>\n

Spark In MapReduce (SIMR) :<\/h4>\n

Another option, besides standalone deployment, is to use SIMR. All the Hadoop users those are not running YARN yet can definitely switch to SIMR. we can simply use SIMR to launch spark jobs inside mapreduce.<\/p>\n

With SIMR, users can start working with spark easily. We can also use its shell within a couple of moments after downloading it. Using SMIR reduces the barrier of deployment, and lets everyone play with spark.<\/p>\n

To understand better we need to dive in launching methods of all three ways one by one:<\/p>\n

1. Launching of Spark in Standalone Mode<\/h4>\n

While we talk about the standalone cluster, Spark supports two deployment modes for it. Such as the cluster mode<\/strong> and the client mode<\/strong>.<\/p>\n

client mode:<\/h5>\n

Driver launch in the same process, in which client submits the application.<\/p>\n

cluster mode:<\/h5>\n

In this mode, driver launch from one of worker node process inside the cluster.<\/p>\n

As it submits the application the client process exits. It exists even without waiting for the application to finish.<\/p>\n

bin\/spark-submit --master spark:\/\/ubuntu:7077 --class com.df.wc.Wordcount ..\/sparkwc.jar \u00a0hdfs:\/\/localhost:9000\/inp hdfs:\/\/localhost:9000\/out0004<\/pre>\n
– Running application in Standalone Mode<\/h5>\n
If we want to run spark application in standalone mode by taking input from HDFS use the code:<\/p>\n
$ .\/bin\/spark-submit --class MyApp.class \u00a0--master MyApp.jar --input hdfs:\/\/localhost:9000\/input-file-path --output output-file-path<\/pre>\n
– Adding the jar<\/h5>\n
If we launch request with spark submit, application jar gets spread over all worker nodes. For any extra jar specify it through \u2013 jars flag, use comma as a delimiter. If it exits with a non-zero code, standalone cluster mode will restart your application.<\/p>\n
2. Launching of Spark on YARN<\/h4>\n
There is a condition that YARN_CONF_DIR points to the directory must contain the (client side) configuration. Basically, these configs are for the hadoop cluster<\/strong>. To write to HDFS and connect to YARN resource manager, we generally use these configs.<\/p>\n
The configs available in this directory will spread to the YARN cluster. So that all containers\u00a0use the same configuration, which is used by an application. As the configuration references, java system properties not managed by YARN.<\/p>\n
Those configs should also be set in the spark application\u2019s configuration.<\/p>\n
There are two deploy modes to launch spark applications while using Yarn. Likewise cluster mode and client mode.<\/p>\n
cluster mode:<\/h5>\n
Inside an application master process, a driver runs. By YARN on the cluster, we can control this process. Also, the client can go away after starting the application.<\/p>\n
client mode:<\/h5>\n
In the client process, the driver runs for requesting resources from YARN, we use application master only.<\/p>\n
In Yarn Mode, we take the address of resource manager from Hadoop configuration. So, the master parameter in here is yarn.<\/p>\n
– If we want to launch spark application in cluster mode. The command is:<\/p>\n
$ .\/bin\/spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options]<\/pre>\n
– If we want to launch spark application in client mode. \u00a0The command is: (replace cluster in above by client)<\/p>\n
$ .\/bin\/spark-submit --class path.to.your.Class --master yarn --deploy-mode client<\/pre>\n
– Debugging Application on YARN<\/h5>\n
Both the application master as well as executors run inside the containers in YARN. After the application has completed YARN has two modes for handling container logs. To understand them in detail let\u2019s look under the hood thoroughly:<\/p>\n
If log aggregation is turned on<\/h5>\n
If log aggregation is turned on while using the YARN. This config enables the container logs. Those logs are replicate as it is to HDFS and then getting erase from the local machine. Later on, if we want to view this file from anywhere on the cluster we can use this command:<\/p>\n
yarn logs -applicationId <app ID><\/pre>\n
While we use this command, it will print the content from all log files from given application.<\/p>\n
To see the container log files in HDFS directly, we can use HDFS shell or API. To find the directory where the log file is available, we will use this command:<\/p>\n
yarn.nodemanager.remote-app-log-dir and \u00a0yarn.nodemanager.remote-app-log-dir-suffix<\/pre>\n
If log aggregation is not turned on<\/h5>\n
While we use YARN if log aggregation isn\u2019t turned on, all the logs are reserve locally on each machine under YARN_APP_LOGS_DIR. Those are \u00a0usually configured to \/tmp\/logs or $HADOOP_HOME\/logs\/userlogs. It only depends on the Hadoop version and installation.<\/p>\n
To view logs for a container, it requires going to the host. Host contains them in their directory. In sub-directories log files are classified by application ID as well as container ID.<\/p>\n
– Adding the jar<\/h5>\n
As we already know, the driver runs on a different machine than the client in cluster mode. So SparkContext.addJar won\u2019t work with files that are local to the client and may out of the box. If we want files available to SparkContext.addJar, add them with –jars in the launch command.<\/p>\n
$ .\/bin\/spark-submit --class my.main.Class \\\n--master yarn \\\n--deploy-mode cluster \\\n--jars my-other-jar.jar,my-other-other-jar.jar \\\nmy-main-jar.jar \\\napp_arg1 app_arg2<\/pre>\n
– If we want to access Spark runtime jars from YARN side, we need to specify-<\/p>\n
spark.yarn.archive or spark.yarn.jars.<\/pre>\n
–\u00a0 Spark will form a zip file with all jar under $SPARK_HOME\/jars, Since, we don’t want to specify. Hence, it will upload it to the distributed cache.<\/p>\n
3. Launching of Spark in MapReduce (SIMR)<\/h4>\n
It allows anyone with access to a Hadoop mapreduce v1 cluster While working on SIMR. Also, helps to run spark out of the box. On top of hadoop mapreduce v1, any user can run spark directly.<\/p>\n
Even it does not require any administrative rights. There is no need to install spark or scala on any of the nodes, to access it. It only requires HDFS access and mapreduce v1. SIMR is open sourced under the apache license<\/strong>.<\/p>\n
In addition, It is beneficial that fundamentally a user can download the package of SIMR (3 files). Those matches their Hadoop cluster and immediately start using spark. Those three files are:<\/p>\n
SIMR runtime script: simr<\/p>\n
simr-<hadoop-version>.jar\n\nspark-assembly-<hadoop-version>.jar<\/pre>\n
It includes interactive spark shell. SMIR allows users to use the shell backed by the computational power of the cluster. Once the SIMR is ready we can simply get it by the command:<\/p>\n
.\/simr --shell:<\/pre>\n
Basically,\u00a0 it requires bundling it along with its dependencies into a jar. It helps to run a spark program. Also, require launching the job through SIMR. For running jobs it uses the following command line syntax:<\/p>\n
.\/simr jar_file main_class parameters<\/pre>\n
– How does SIMR work<\/h4>\n
The user can interact with the driver program by using SIMR. Moreover, SIMR runs the relay server on the master mapper<\/strong>. It runs the relay client on the machine which launches SIMR.<\/p>\n
When input and output come from client and driver, it goes to and fro between the client and the master mapper. Hence, to achieve all this HDFS is greatly used.<\/p>\n
Conclusion<\/h3>\n
Hadoop spark compatibility tutorial said that Hadoop needs a diverse processing engine to process its data, on the other hand, spark needs a reliable storage system which is fault-tolerant as well as distributed. Hence for Big Data professionals its the best combination to solve Big Data problems more efficiently than ever.<\/p>\n","protected":false},"excerpt":{"rendered":"
This tutorial is all about\u00a0Hadoop Spark Compatibility. Hadoop and Spark together build a very powerful system to address all the Big Data requirements. Spark complements Hadoop with tons of power, you can handle all...<\/p>\n","protected":false},"author":1,"featured_media":73259,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[778,779,780,781,782,783,784],"class_list":["post-2017","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-apache-spark-and-hadoop-working-together","tag-apache-spark-compatibility-with-hadoop","tag-apache-spark-with-hadoop-why-it-matters","tag-compatibility-between-spark-and-hadoop","tag-launching-of-spark-in-mapreduce","tag-launching-of-spark-in-standalone-mode","tag-launching-of-spark-on-yarn"],"yoast_head":"\nHadoop Spark Compatibility: Hadoop+Spark better together - TechVidvan<\/title>\n<meta name=\"description\" content=\"Hadoop Spark Compatibility: how spark complements hadoop to handle diverse workloads, what are the different ways to launch spark jobs on hadoop: Standalone, YARN, SIMR\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Hadoop Spark Compatibility: Hadoop+Spark better together - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Hadoop Spark Compatibility: how spark complements hadoop to handle diverse workloads, what are the different ways to launch spark jobs on hadoop: Standalone, YARN, SIMR\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-09T13:09:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-compatible-with-Hadoop-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Hadoop Spark Compatibility: Hadoop+Spark better together - TechVidvan","description":"Hadoop Spark Compatibility: how spark complements hadoop to handle diverse workloads, what are the different ways to launch spark jobs on hadoop: Standalone, YARN, SIMR","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/","og_locale":"en_US","og_type":"article","og_title":"Hadoop Spark Compatibility: Hadoop+Spark better together - TechVidvan","og_description":"Hadoop Spark Compatibility: how spark complements hadoop to handle diverse workloads, what are the different ways to launch spark jobs on hadoop: Standalone, YARN, SIMR","og_url":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2018-01-09T13:09:53+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-compatible-with-Hadoop-1.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Hadoop Spark Compatibility: Hadoop+Spark better together","datePublished":"2018-01-09T13:09:53+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/"},"wordCount":1643,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-compatible-with-Hadoop-1.jpg","keywords":["Apache Spark and Hadoop: Working Together","Apache spark compatibility with Hadoop","Apache Spark with Hadoop \u2013 Why it Matters?","Compatibility between Spark and Hadoop","Launching of Spark in MapReduce","Launching of Spark in Standalone Mode","Launching of Spark on YARN"],"articleSection":["Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/","url":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/","name":"Hadoop Spark Compatibility: Hadoop+Spark better together - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-compatible-with-Hadoop-1.jpg","datePublished":"2018-01-09T13:09:53+00:00","description":"Hadoop Spark Compatibility: how spark complements hadoop to handle diverse workloads, what are the different ways to launch spark jobs on hadoop: Standalone, YARN, SIMR","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-compatible-with-Hadoop-1.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-compatible-with-Hadoop-1.jpg","width":1200,"height":628,"caption":"Spark compatible with Hadoop"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/hadoop-spark-compatibility\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Hadoop Spark Compatibility: Hadoop+Spark better together"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2017","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=2017"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2017\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73259"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=2017"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=2017"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=2017"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}