{"id":2012,"date":"2018-01-08T12:27:43","date_gmt":"2018-01-08T12:27:43","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=669"},"modified":"2018-01-08T12:27:43","modified_gmt":"2018-01-08T12:27:43","slug":"spark-lazy-evaluation","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/","title":{"rendered":"Apache Spark Lazy Evaluation: In Spark RDD"},"content":{"rendered":"

In this blog, we will capture one of the important features of RDD, Spark Lazy Evaluation. Spark RDD (Resilient Distributed Datasets), <\/strong>collect all the elements of data in the cluster which are partitioned.\u00a0Its a group of immutable objects arranged in the cluster in a distinct manner.<\/p>\n

Here, we will discuss spark lazy evaluation, reasons for carrying lazy evaluation technique, how the spark is managing with this technique at last advantages of spark lazy evaluation.<\/p>\n

Introduction<\/h3>\n

What is lazy evaluation-<\/p>\n

\u201cLAZY\u201d the word itself indicates its meaning \u2018not at the same time<\/em>\u2019.<\/p>\n

That means, it evaluates something only when we require it. In accordance with a spark, it does not execute each operation right away, that means it does not start until we trigger any action. Once an action is called all the transformations will execute in one go.<\/p>\n

This process is more appealing than executing one operation at the same time. Execution at the same time may lead to delay the process. Overcoming to the downtime, it gives birth to Lazy Evaluation<\/strong>.<\/p>\n

Using lazy evaluation, It figured out that all transformations can be combined together into a single transformation and executed together. Simply, it says while performing any operation it will not execute until we trigger an action.<\/p>\n

We need to call an action every time we want to execute the process. Especially while we perform any transformation, it does not execute immediately.<\/p>\n

DAG in Spark<\/h4>\n

Through DAG, Spark maintains the record of every operation performed, DAG refers to Directed Acyclic Graph<\/strong>. That keeps the track of each step through its arrangement of vertices and edges.<\/p>\n

So, when we call any action it strikes to DAG graph directly and DAG keeps maintenance of operations which triggers the execution process forward.<\/p>\n

If we talk about Hadoop mapreduce, we need to minimize the steps of mapreduce passes. That is only possible by clubbing the operations together. While in spark DAG graph already maintains the records of all the operations, it clubs all the operations in one.<\/p>\n

Rather than making single execution graph for each operation. Thus it creates a difference between Hadoop mapreduce and apache spark.<\/p>\n

So, in this way by waiting until last minute to execute our code will also enhance the performance. It also boosts the efficiency of the process over the cluster.<\/p>\n

Advantages of\u00a0Spark Lazy Evaluation<\/h3>\n

There are several perks of lazy evaluation listed below:<\/p>\n

\"Spark<\/a><\/h3>\n

1. Reduces Complexities<\/h4>\n

As we know that, we are not executing every operation rather than we are executing the entire process only once. It allows us to work with an infinite data structure.<\/p>\n

An action is only triggered when it is required. It results in saving the time as well as reducing the space complexity.<\/p>\n

2. Optimization<\/h4>\n

Due to lazy evolution, the system works more efficiently with fewer resources. That means, it also decreases the number of queries.<\/p>\n

3. Develops Manageability<\/h4>\n

By manageability, we signifying the organizing manner of a program. That organize it into smaller operations which enhances the efficiency. It also helps in reduces the number of passes on data by grouping the operations.<\/p>\n

4. Saves Computation and increases speed<\/h4>\n

Due to this process, we are not performing every calculation at the instance. This is saving us from the bundle of calculations we owe. Now we only need to calculate important values, that saves our time and also speeds up the process.<\/p>\n

Conclusion<\/h3>\n

As a result, we have seen spark lazy evaluation plays the very important role in the execution process and had several benefits of it. It enhances the efficiency of the system by reducing a number of executions of operations.<\/p>\n

This feature also uplift the performance of the system. It also maintains the track of operations through DAG which optimizes the process. Lazy Evaluation works as a key building block in operations of Spark RDD.<\/p>\n","protected":false},"excerpt":{"rendered":"

In this blog, we will capture one of the important features of RDD, Spark Lazy Evaluation. Spark RDD (Resilient Distributed Datasets), collect all the elements of data in the cluster which are partitioned.\u00a0Its a...<\/p>\n","protected":false},"author":1,"featured_media":73275,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[733,734,735,736,737,738,739],"class_list":["post-2012","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-dag-directed-acyclic-graph","tag-lazy-evaluation-in-apache-spark-a-quick-guide","tag-lazy-evaluation-rdd-spark","tag-reasons-for-carrying-lazy-evaluation-technique","tag-what-are-the-benefits-of-lazy-evaluation-in-rdd-in-apache-spark","tag-what-is-the-reason-behind-keeping-lazy-evaluation-in-apache-spark","tag-why-its-lazy-and-what-is-the-advantage"],"yoast_head":"\nApache Spark Lazy Evaluation: In Spark RDD - TechVidvan<\/title>\n<meta name=\"description\" content=\"Spark Lazy Evaluation tutorial covers lazy evaluation introduction & benefits of lazy evaluation along with introduction to DAG in spark.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Spark Lazy Evaluation: In Spark RDD - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Spark Lazy Evaluation tutorial covers lazy evaluation introduction & benefits of lazy evaluation along with introduction to DAG in spark.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-08T12:27:43+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-Lazy-Evaluation-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Spark Lazy Evaluation: In Spark RDD - TechVidvan","description":"Spark Lazy Evaluation tutorial covers lazy evaluation introduction & benefits of lazy evaluation along with introduction to DAG in spark.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/","og_locale":"en_US","og_type":"article","og_title":"Apache Spark Lazy Evaluation: In Spark RDD - TechVidvan","og_description":"Spark Lazy Evaluation tutorial covers lazy evaluation introduction & benefits of lazy evaluation along with introduction to DAG in spark.","og_url":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2018-01-08T12:27:43+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-Lazy-Evaluation-01.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Apache Spark Lazy Evaluation: In Spark RDD","datePublished":"2018-01-08T12:27:43+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/"},"wordCount":640,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-Lazy-Evaluation-01.jpg","keywords":["DAG (Directed Acyclic Graph)","Lazy Evaluation in Apache Spark \u2013 A Quick guide","Lazy Evaluation: RDD Spark","reasons for carrying lazy evaluation technique","What are the benefits of lazy evaluation in RDD in Apache Spark","What is the reason behind keeping lazy evaluation in Apache Spark","Why its lazy and what is the advantage"],"articleSection":["Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/","url":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/","name":"Apache Spark Lazy Evaluation: In Spark RDD - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-Lazy-Evaluation-01.jpg","datePublished":"2018-01-08T12:27:43+00:00","description":"Spark Lazy Evaluation tutorial covers lazy evaluation introduction & benefits of lazy evaluation along with introduction to DAG in spark.","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-Lazy-Evaluation-01.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-Lazy-Evaluation-01.jpg","width":1200,"height":628,"caption":"Spark Lazy Evaluation"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/spark-lazy-evaluation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Lazy Evaluation: In Spark RDD"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2012","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=2012"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2012\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73275"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=2012"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=2012"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=2012"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}