{"id":2035,"date":"2018-01-19T06:51:11","date_gmt":"2018-01-19T06:51:11","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=874"},"modified":"2018-01-19T06:51:11","modified_gmt":"2018-01-19T06:51:11","slug":"spark-executor","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/","title":{"rendered":"Apache Spark Executor &#8211; For Executing Tasks"},"content":{"rendered":"<p>There are some distributed agents in spark, which are responsible for executing tasks, those distributed agents are <strong>Spark Executor<\/strong>. In this blog, we will learn the whole concept of Apache spark executor.<\/p>\n<p>Moreover, we will learn how we can create executor instance in spark, launch task method in spark executor and how we can stop spark executor and request for the report.<\/p>\n<h3>What Is Spark Executor<\/h3>\n<p>There are some distributed agents in spark, which are responsible for executing tasks. Those distributed agents are e<em>xecutors<\/em>. There are some conditions in which we create executor, such as:<\/p>\n<ul>\n<li>When CoarseGrainedExecutorBackend receives RegisteredExecutor message. Only for Spark Standalone and YARN.<\/li>\n<li>While Mesos\u2019s MesosExecutorBackend registered on spark.<\/li>\n<li>When LocalEndpoint is created for local mode.<\/li>\n<\/ul>\n<p>It runs for the complete lifespan of a spark application, i.e. static allocation of an executor. Although, we can also prefer for dynamic allocation.<\/p>\n<p>In other words, for an application, a process launched on a worker node. Moreover, it runs tasks and also keeps data in disk storage or in memory among them. Basically, each application has its own executor.<\/p>\n<p>In addition, it reports partial metrics for active tasks to the receiver on the driver. It offers in-memory storage for RDDs. Those are cached in spark applications by block manager. While an executor initiates, it registers with the driver. Moreover, it directly communicates to execute tasks.<\/p>\n<p>By executor id and the host on which an executor runs executor offers are described. It can run multiple tasks over its lifetime, both sequentially and parallelly. Also, tracks running tasks by their task ids in runningTasks internal registry. For launching tasks, executors use an executor task launch worker thread pool.<\/p>\n<p>Moreover,\u00a0 it sends metrics and heartbeats by using <strong>Heartbeat Sender Thread<\/strong>. It is possible to have as many spark executors as data nodes, also can have as many cores as you can get from the cluster mode. We can describe executors by their id, hostname, environment (as SparkEnv), and classpath.<\/p>\n<p><b>Note<\/b>: Executor backends exclusively manage executors.<\/p>\n<h3>Creating Spark\u00a0Executor Instance<\/h3>\n<p>We can create executor with the help of \u00a0following:<\/p>\n<ul>\n<li>Executor ID.<\/li>\n<li>To access the local MetricsSystem and block manager we use SparkEnv. We can also use it to access the local serializer.<\/li>\n<li>Executor\u2019s hostname.<\/li>\n<li>To add to tasks&#8217; classpath, a collection of user-defined JARs. By default, it is empty.<\/li>\n<li>Flag whether it runs in local or cluster mode (disabled by default, i.e. cluster is preferred).<\/li>\n<\/ul>\n<p>While created, the following INFO messages appear in the logs:<\/p>\n<p><strong>INFO Executor: Starting executor ID [executorId] on host [executorHostname]<\/strong><\/p>\n<h3>Launching Task \u2014 launch Task Method<\/h3>\n<div id=\"attachment_73190\" style=\"width: 1210px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2019\/11\/Launching-tasks-on-executor-using-TaskRunners-01.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-73190\" class=\"wp-image-73190 size-full\" src=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2019\/11\/Launching-tasks-on-executor-using-TaskRunners-01.jpg\" alt=\"Spark Executor - Launching Tasks\" width=\"1200\" height=\"628\" \/><\/a><p id=\"caption-attachment-73190\" class=\"wp-caption-text\">Spark Executor &#8211; Launching tasks on executor using TaskRunner<\/p><\/div>\n<p>This method executes the input serializedTask task concurrently.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">launchTask(\n\u00a0context: ExecutorBackend,\n\u00a0taskId: Long,\n\u00a0attemptNumber: Int,\n\u00a0taskName: String,\n\u00a0serializedTask: ByteBuffer): Unit<\/pre>\n<p>LaunchTask creates a TaskRunner, internally. Afterwards, registers it in runningTasks internal registry with the help of taskId. Ultimately, executes it on &#8220;executor task launch worker&#8221; thread pool.<\/p>\n<h3>&#8220;Executor task launch worker&#8221; Thread Pool \u2014 ThreadPool Property<\/h3>\n<p>To launch, It uses threadPool daemon cached thread pool by task launch worker id. Generally, when the executor is created, threadPool is created and shut down when it stops.<\/p>\n<h3>Stopping Executor \u2014 Stop Method<\/h3>\n<p>This method stops requests MetricsSystem for a report.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">stop(): Unit<\/pre>\n<p>Stop method uses SparkEnv to access the current MetricsSystem. The method stop() shuts driver-heartbeater thread down then waits at most 10 seconds. It shuts executor task launch worker thread pool down. Although, when not local, it requests SparkEnv to stop.<\/p>\n<p><strong>Note:<\/strong>\u00a0We use stop() while CoarseGrainedExecutorBackend and LocalEndpoint stop their managed executors.<\/p>\n<h3><b>Conclusion<\/b><\/h3>\n<p>In this blog, we have learned, the whole concept of Spark executors. Also, we have seen how executors are helpful for executing tasks. Moreover, we can have as many executors we want. Hence, it enhances the performance of the system.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There are some distributed agents in spark, which are responsible for executing tasks, those distributed agents are Spark Executor. In this blog, we will learn the whole concept of Apache spark executor. Moreover, we&#46;&#46;&#46;<\/p>\n","protected":false},"author":1,"featured_media":73166,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[946,947,948,949,950,951,952],"class_list":["post-2035","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-apache-spark-executors-for-executing-tasks","tag-execute-tasks","tag-executor","tag-executor-in-apache-spark","tag-spark-executor","tag-what-are-exceutors","tag-what-is-executor-in-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Apache Spark Executor - For Executing Tasks - TechVidvan<\/title>\n<meta name=\"description\" content=\"Apache Spark executor- what is spark executor, creating instance in spark executor, launching spark method, stopping executor in spark\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/spark-executor\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Spark Executor - For Executing Tasks - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Apache Spark executor- what is spark executor, creating instance in spark executor, launching spark method, stopping executor in spark\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/spark-executor\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-19T06:51:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HeartbeatReceivers-Heartbeat-Message-Handler-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Spark Executor - For Executing Tasks - TechVidvan","description":"Apache Spark executor- what is spark executor, creating instance in spark executor, launching spark method, stopping executor in spark","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/","og_locale":"en_US","og_type":"article","og_title":"Apache Spark Executor - For Executing Tasks - TechVidvan","og_description":"Apache Spark executor- what is spark executor, creating instance in spark executor, launching spark method, stopping executor in spark","og_url":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2018-01-19T06:51:11+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HeartbeatReceivers-Heartbeat-Message-Handler-01.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Apache Spark Executor &#8211; For Executing Tasks","datePublished":"2018-01-19T06:51:11+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/"},"wordCount":628,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HeartbeatReceivers-Heartbeat-Message-Handler-01.jpg","keywords":["Apache Spark Executors: for executing tasks","execute tasks","Executor","Executor in Apache Spark","Spark Executor","what are Exceutors","what is Executor in Spark?"],"articleSection":["Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/spark-executor\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/","url":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/","name":"Apache Spark Executor - For Executing Tasks - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HeartbeatReceivers-Heartbeat-Message-Handler-01.jpg","datePublished":"2018-01-19T06:51:11+00:00","description":"Apache Spark executor- what is spark executor, creating instance in spark executor, launching spark method, stopping executor in spark","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/spark-executor\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HeartbeatReceivers-Heartbeat-Message-Handler-01.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/HeartbeatReceivers-Heartbeat-Message-Handler-01.jpg","width":1200,"height":628,"caption":"Apache Spark Executor- To Execute Tasks"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/spark-executor\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Executor &#8211; For Executing Tasks"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2035","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=2035"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2035\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73166"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=2035"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=2035"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=2035"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}