{"id":643,"date":"2018-01-08T12:22:27","date_gmt":"2018-01-08T12:22:27","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=643"},"modified":"2018-01-08T12:22:27","modified_gmt":"2018-01-08T12:22:27","slug":"sparkcontext-entry-point-spark","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/","title":{"rendered":"SparkContext In Apache Spark: Entry Point to Spark Core"},"content":{"rendered":"

This tutorial gives information on the main entry point to spark core i.e. Apache Spark SparkContext. Apache Spark is a powerful cluster computing engine, therefore, it is designed for fast computation of big data.<\/p>\n

Here, we will learn what is Apache Spark SparkContext. Furthermore, we will discuss the process to create SparkContext Class in Spark and the facts that how to stop SparkContext in Spark.<\/p>\n

To get this concept deeply, we will also study various functions of SparkContext in Spark.<\/p>\n

\"SparkContext<\/a><\/p>\n

Introduction<\/h3>\n

We can say Apache Spark SparkContext is a heart<\/em> of spark application. It is the Main entry point to Spark Functionality<\/strong>. Generating, SparkContext is a most important task for Spark Driver Application and set up internal services and also constructs a connection to Spark execution environment.<\/p>\n

This is essentially a client of Spark\u2019s execution environment, that acts as a master of Spark Application.<\/p>\n

Moreover, once we create Apache Spark SparkContext we can use it in following ways. We can make RDDs (Resilient distributed datasets) from it, which is also used to obtain broadcast variables.<\/p>\n

Broadcast variables<\/strong> mean, which allow the programmer to cache a read-only variable and allows reading on each machine than shipping copy of it with jobs. Furthermore, we can create accumulator<\/em> using SparkContext. It allows to access spark services and helps to run jobs until we stop SparkContext by own.<\/p>\n

In addition, SparkContext allows accessing spark cluster manager, spark Standalone, Yarn, and Apache Mesos any one of them can be selected as a cluster manager.<\/p>\n

If we use YARN (head node), and node manager (worker node), it may work to allocate<\/span> containers for executors. As resources are available then, it will allocate<\/span> memory and cores.<\/p>\n

It distributes according to the configuration. If we use sparks manager (head node), and spark slave (worker node) will work to allocate<\/span> resources.<\/p>\n

SparkContext – Ways to create Class<\/h3>\n

The first step to create Apache Spark SparkContext.\u00a0It is the main entry point to spark functionality, is to make sparkconf<\/strong>. We have some configuration parameters. Those parameters we pass to sparkcontext through spark driver application and these parameters explain the properties of the application.<\/p>\n

We configure the parameters according to functionalities we need. Spark use some of them to allocate resources on a cluster by executors. Resources like cluster, like the number, memory size, and cores.<\/p>\n

In addition, it helps us to access spark cluster in a better way. Due to sparkcontext objects, we can use functions such as textfile, sequence file and parallelize. There are following context in which it can run are local, yarn-client, Mesos URL and Spark URL.<\/p>\n

As we discussed earlier, once we create sparkcontext. We can use it to create RDDs, broadcast variable, accumulator, and run jobs. Until we stop sparkcontext we can carry most of all these things.<\/p>\n

SparkContext – How to stop<\/h3>\n

There is a condition that only a single sparkcontext can be active for single JVM<\/strong>. Before creating a new sparkcontext we need to stop () the active one.<\/p>\n

To Stop()<\/b><\/p>\n

stop(): Unit<\/pre>\n
While we apply this command we get a message written below:<\/p>\n
INFO SparkContext: Successfully stopped SparkContext<\/i><\/strong><\/p>\n
This message signifies that running sparkcontext stopped successfully.<\/p>\n
SparkContext – Various tasks
\n<\/b><\/h3>\n
SparkContext offers following functions. Those are list-up below:<\/p>\n
\"Spark<\/a><\/p>\n
1. We have following functions to get the current status of Spark Application<\/h4>\n
– SpkEnv<\/strong><\/p>\n
SparkENV is a runtime environment with spark\u2019s public services<\/strong>.\u00a0 For a running spark application, it holds all the running services objects.<\/p>\n
It may also include serializer,\u00a0 block manager, map output tracker, Akka actor system and also interacts to establish a distributed computing platform for spark application.<\/p>\n
– To access SparkEnv<\/strong><\/p>\n
SparkEnv.get<\/p>\n
– SparkConf <\/strong><\/p>\n
In\u00a0spark, each application is configured separately with spark properties. Properties also manage all the applications. We can also set these properties manually on sparkconf.<\/p>\n
We can configure following common properties, such as master URL, application name, arbitrary key-value pair and configured through set method.<\/p>\n
– Deployment environment (as master URL)<\/strong><\/p>\n
We can categorize deployment environment in two ways, such as local and clustered. We can also run spark in local mode. This is non-distributed single – JVM deployment mode. All the execution components are present in same single JVM. It includes driver, executor, local scheduler backend, and master.<\/p>\n
Hence,\u00a0a local mode is the only mode where drivers are useful for execution. This mode is very appropriate for testing, debugging.<\/p>\n
Also, helps in demonstration purposes. Due to there is no earlier setup require to launch spark applications. Local mode is also called as spark in-process<\/strong> or a local version of Spark<\/strong>.<\/p>\n
2 Spark in Cluster mode<\/h4>\n
Spark can run in cluster mode also. The following cluster managers (task schedulers or resource managers) are currently supported:<\/p>\n
1.Spark\u2019s own built-in standalone cluster manager<\/p>\n
2.Hadoop YARN<\/p>\n
3.Apache Mesos<\/p>\n
– Application name<\/strong><\/p>\n
This function gives the value to the mandatory spark.app.name setting.<\/p>\n
– Unique identifier of the execution attempt<\/strong><\/p>\n
This application provides the unique identifier of a spark application.<\/p>\n
– Deploy Mode<\/strong><\/p>\n
This mode returns the current value of spark.submit.deployMode setting or client if not set.<\/p>\n
– Default level of parallelism<\/strong><\/strong><\/p>\n
When we create Spark RDDs without specifying the number by a user. That specifies the number of partitions in Spark RDDs.<\/p>\n
– SparkUser<\/strong><\/p>\n
It is the user that initiated the sparkcontext instance.<\/p>\n
3. We have following functions to set the configuration<\/h4>\n
– Master URL<\/strong><\/p>\n
This method helps to returns back the current value of spark.master. That is deployment environment in use.<\/p>\n
– Local properties – Creating Logical Job Groups <\/strong><\/p>\n
To form logical groups of jobs by means of properties, we use local properties. These create job launched from the different thread. That belongs to the single logic group. We can also set local properties. That may affect spark Jobs submitted from the thread, as spark fair scheduler pool.<\/p>\n
We can also use our own custom properties. Properties can propagate through to worker tasks. We can access thereby TaskContext.getLocalProperty.<\/p>\n
–\u00a0Default Logging level<\/strong><\/p>\n
In a spark application, default logging level helps to set the root login level.<\/p>\n
4. We have following functions to access various services<\/h4>\n
– Task Scheduler<\/strong><\/p>\n
It helps to manage (i.e. reads or writes) task scheduler internal property.<\/p>\n
– LiveListenerBus<\/strong><\/p>\n
We use livelistnerbus to declare application-wide events in a spark application. It makes a path to pass listener events to register spark listeners.<\/p>\n
– Block Manager<\/strong><\/p>\n
It is a key-value store for blocks of data in spark. Also, acts as a local cache. \u00a0It runs on every “node” in a spark, i.e. the driver and executors.<\/p>\n
– Scheduler Backend <\/strong><\/p>\n
To support various cluster managers, it works as a pluggable interface. The first one is Apache Mesos<\/strong>, second is Hadoop YARN<\/strong> and the third one is\u00a0 Spark Standalone<\/strong>.<\/p>\n
– Shuffle Manager<\/strong><\/p>\n
It is the pluggable mechanism for shuffle systems. On the driver and executors, it tracks shuffle dependencies for ShuffleMapStage.<\/p>\n
5. Provides a function To Cancel a job<\/h4>\n
It simply means to cancel a Spark job (with an optional reason) by requesting \u00a0DAG Scheduler.<\/p>\n
6. Provides a function Closure Cleaning<\/h4>\n
Whenever we call an action, Spark cleans up the closure. That is the body of the action before it is serialized. This method is closure cleaning method. In addition, it not only cleans the closure but also does it transitively. That means referenced closures are cleaned transitively.<\/p>\n
7. Provides a function To cancel a stage<\/h4>\n
It simply means to cancel a Spark stage (with an optional reason). Actually, by requesting DAGScheduler.<\/p>\n
8. Provides a function To Register Spark listener<\/h4>\n
We can register a custom Spark Listener Interface by using add Spark Listener method.<\/p>\n
9. It allows Programmable Dynamic allocation<\/h4>\n
SparkContext offers the several methods as \u00a0API developer for dynamic allocation of executors:<\/p>\n
    \n
  1. Request Executors<\/li>\n
  2. Kill Executors<\/li>\n
  3. Request Total Executors<\/li>\n
  4. GetExecutorIds<\/li>\n<\/ol>\n
    10. It provides a function To access persistent RDD<\/h4>\n
    This provides the collection of spark RDDs. Those have considered themselves as persistent by using the cache.<\/p>\n
    11. It provides a function to un-persist RDDs<\/h4>\n
    Unpersist removes the RDD from the master\u2019s block manager internal persistent RDD\u00a0pamapping.<\/p>\n
    Conclusion<\/h3>\n
    Hence, we have discussed, an introduction of\u00a0SparkContext,\u00a0a creation of\u00a0SparkContext, how to stop it and it’s various tasks. However, provides the various functions in sparks, such as getting the current status, set the configuration, cancel a job, cancel a stage and much more.<\/p>\n
    As a result, we get SparkContext is the main entry point to spark core and it acts as a heart of a spark application.<\/p>\n","protected":false},"excerpt":{"rendered":"
    This tutorial gives information on the main entry point to spark core i.e. Apache Spark SparkContext. Apache Spark is a powerful cluster computing engine, therefore, it is designed for fast computation of big data....<\/p>\n","protected":false},"author":1,"featured_media":73260,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[676,677,678,679,680,681,682,683],"class_list":["post-643","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-apache-spark-sparkcontext","tag-create-sparkcontext","tag-sparkcontext","tag-sparkcontext-example","tag-sparkcontext-entry-point-to-spark-core","tag-what-is-sparkconf","tag-what-is-sparkconf-in-spark","tag-what-is-sparkcontext-in-apache-spark"],"yoast_head":"\nSparkContext In Apache Spark: Entry Point to Spark Core - TechVidvan<\/title>\n<meta name=\"description\" content=\"SparkContext in Apache Spark-heart of Spark application.Learn Introduction of SparkContext,sparkconf,how to create SparkContext,functions,Spark cluster mode\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SparkContext In Apache Spark: Entry Point to Spark Core - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"SparkContext in Apache Spark-heart of Spark application.Learn Introduction of SparkContext,sparkconf,how to create SparkContext,functions,Spark cluster mode\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-08T12:22:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/SparkContext-in-Apache-Spark-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SparkContext In Apache Spark: Entry Point to Spark Core - TechVidvan","description":"SparkContext in Apache Spark-heart of Spark application.Learn Introduction of SparkContext,sparkconf,how to create SparkContext,functions,Spark cluster mode","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/","og_locale":"en_US","og_type":"article","og_title":"SparkContext In Apache Spark: Entry Point to Spark Core - TechVidvan","og_description":"SparkContext in Apache Spark-heart of Spark application.Learn Introduction of SparkContext,sparkconf,how to create SparkContext,functions,Spark cluster mode","og_url":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2018-01-08T12:22:27+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/SparkContext-in-Apache-Spark-01.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"SparkContext In Apache Spark: Entry Point to Spark Core","datePublished":"2018-01-08T12:22:27+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/"},"wordCount":1410,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/SparkContext-in-Apache-Spark-01.jpg","keywords":["Apache Spark SparkContext","create sparkcontext","SparkContext","sparkcontext example","SparkContext: Entry Point to Spark Core","what is sparkconf","what is sparkconf in spark","what is SparkContext in Apache Spark"],"articleSection":["Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/","url":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/","name":"SparkContext In Apache Spark: Entry Point to Spark Core - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/SparkContext-in-Apache-Spark-01.jpg","datePublished":"2018-01-08T12:22:27+00:00","description":"SparkContext in Apache Spark-heart of Spark application.Learn Introduction of SparkContext,sparkconf,how to create SparkContext,functions,Spark cluster mode","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/SparkContext-in-Apache-Spark-01.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/SparkContext-in-Apache-Spark-01.jpg","width":1200,"height":628,"caption":"SparkContext in Apache Spark"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/sparkcontext-entry-point-spark\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"SparkContext In Apache Spark: Entry Point to Spark Core"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/643","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=643"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/643\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73260"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=643"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=643"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=643"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}