{"id":659,"date":"2018-01-05T12:26:29","date_gmt":"2018-01-05T12:26:29","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=659"},"modified":"2018-01-05T12:26:29","modified_gmt":"2018-01-05T12:26:29","slug":"spark-rdd-features","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/","title":{"rendered":"Spark RDD – Features, Limitations and Operations"},"content":{"rendered":"

Apache Spark RDD makes developer’s\u00a0\u00a0work more efficient, as it divides cluster into nodes to compute parallel operations on each node. Before anything else, we will go through the brief introduction of Spark RDD.<\/p>\n

After that, we will start looking for sparkling features of Spark RDD. looking forward we will study about operations on RDD as well as limitations to Spark RDD usage.<\/p>\n

Introduction to Apache Spark RDD<\/h3>\n

Apache Spark RDDs (Resilient Distributed Datasets<\/strong>) are a basic abstraction of spark which is immutable<\/em>. These are logically partitioned that we can also apply parallel operations on them.<\/p>\n

Spark RDDs give power to users to control them. Above all, users may also persist an RDD in memory.\u00a0 Also, can reuse RDD efficiently across the parallel operation. Apache Spark RDD is the read-only partitioned collection of records.<\/p>\n

There are two ways to create RDDs –<\/p>\n

1. Parallelize the present collection in our dataset<\/p>\n

2. Referencing a dataset in the external storage system.<\/p>\n

Prominent Features<\/h3>\n

There are following traits of Resilient distributed datasets. Those are list-up below:<\/p>\n

\"Spark<\/a><\/p>\n

1. In-Memory<\/h4>\n

It is possible to store data in spark RDD. Storing of data in spark RDD is size as well as quantity independent. We can store as much data we want in any size. In-memory computation means operate information in the main random access memory.<\/p>\n

It requires operating across jobs, not in complicated databases. Since operating jobs in databases slow the drive.<\/p>\n

2. Lazy Evaluations<\/h4>\n

By its name, it says that on calling some operation, execution process doesn\u2019t start instantly. To trigger the execution, an action is a must. Since that action takes place, data inside RDD cannot get transform or available.<\/p>\n

Through DAG, Spark maintains the record of every operation performed. DAG refers to Directed Acyclic Graph<\/strong>.<\/p>\n

3. Immutable and Read-only<\/h4>\n

Since, RDDs are immutable, which means unchangeable<\/em> over time. That property helps to maintain consistency when we perform further computations.<\/p>\n

As we can not make any change in RDD once created, it can only get transformed into new RDDs. This is possible through its transformations processes.<\/p>\n

4. Cacheable or Persistence<\/h4>\n

We can store all the data in persistent storage, memory, and disk. Memory (most preferred) and disk (less Preferred because of its slow access speed). We can also extract it directly from memory.<\/p>\n

Hence this property of RDDs makes them useful for fast computations. Therefore, we can perform multiple operations on the same data. Also, leads reusability which also helps to compute faster.<\/p>\n

5. Partitioned<\/h4>\n

Each dataset is logically partitioned and distributed across nodes over the cluster. They are just partitioned to enhance the processing, Not divided internally. This arrangement of partitions provides parallelism.<\/p>\n

6. Parallel<\/h4>\n

As we discussed earlier, RDDs are logically partitioned over the cluster. While we perform any operations, it executes parallelly on entire data.<\/p>\n

7. Fault Tolerance<\/h4>\n

While working on any node, if we lost any RDD itself recovers itself. When we apply different transformations on RDDs, it creates a logical execution plan. The logical execution plan is generally known as lineage graph.<\/p>\n

As a consequence, we may lose RDD as if any fault arises in the machine. So by applying the same computation on that node of the lineage graph, we can recover our same dataset again. As a matter of fact, this process enhances its property of Fault Tolerance.<\/p>\n

8. Location Stickiness<\/h4>\n

RDDs supports placement preferences. That refers information of the location of RDD. That DAG(Directed Acyclic Graph)\u00a0 scheduler use to place computing partitions on.<\/p>\n

DAG helps to manage the tasks as much close to the data to operate efficiently. This placing of data also enhances the speed of computations.<\/p>\n

9. Typed<\/h4>\n

We have several types of RDDs which are: RDD [long], RDD [int], RDD [String] .<\/p>\n

10. Coarse-grained Operations<\/h4>\n

RDDs support coarse-grained operations. That means we can perform an operation on entire cluster once at a time.<\/p>\n

11. No-Limitations<\/h4>\n

There is no specific number that limits the usage of RDD. We can use as much RDDs we require. It totally depends on the size of its memory or disk.<\/p>\n

Spark RDD – Operations<\/h3>\n

As we discussed earlier the immutable nature of RDDs. We know that RDDs are unable to change with time, this immutable nature creates confusion to us that if we can not change it so how are we gonna use it?<\/p>\n

Yes, we can not change it but we can modify RDDs as per our requirements. \u00a0We can perform different operations on RDD as well as on data storage to form another RDDs from it. There are two different operations:<\/p>\n

\"Spark<\/a>

Spark RDD – Transformation and Action<\/p><\/div>\n

    \n
  1. TRANSFORMATION<\/li>\n
  2. ACTION<\/li>\n<\/ol>\n

    To modify the available datasets, we need to provide step by step instructions to the spark. Those steps must clearly explain what changes we want. That set of instructions is generally known as \u201cTransformations\u201d.<\/p>\n

    Transformations are operations on RDDs which results in new RDD such as Map, Filter.<\/p>\n

    Actions are operations, triggers the process by returning the result back to program. Transformation and actions work differently.<\/p>\n

    We can judge that which operations take place by its results. Transformation returns RDDs and actions return some other data types.<\/p>\n

    1. Transformation<\/h4>\n

    Transformation is a process of forming new RDDs from the existing ones. Transformation is a user specific function. It is a process of changing the current dataset in the dataset we want to have.<\/p>\n

    Also, can create any number of RDDs we want. We do not change the current RDD as we know they are immutable. So we can produce more RDDs out of it by applying several computations. Some common transformations supported by Spark are:<\/p>\n

    For example,\u00a0 Map(func), Filter(func), Mappartitions (func), Flatmap (func) etc.<\/p>\n

    All transformed RDDs are lazy in nature. As we are already familiar with the term \u201cLazy Evaluations\u201d. That means it does not produce their results instantly. However, we always require an action to complete the computation.<\/p>\n

    To trigger the execution, an action is a must. Up to that action data inside RDD is not transformed or available.<\/p>\n

    After transformation, you incrementally build the lineage. That lineage is which formed by all the parent RDDs of final RDDs. As soon as the execution process ends,\u00a0 resultant RDDs will be completely different from their parent RDDs.<\/p>\n

    They can be smaller (e.g. filter, count, distinct, sample), bigger (e.g. flatMap, union, Cartesian) or the same size (e.g. map).<\/p>\n

    Transformation can categorize further as: Narrow Transformations, Wide Transformations.<\/p>\n

    a. Narrow Transformations<\/h5>\n

    Narrow transformations are the result of a map, filter. As such that is from the data from a single partition only. That signifies it is self-sustained.<\/p>\n

    An output RDD also has partitions with records.\u00a0 In the parent RDD, that output originates from a single partition. Additionally,\u00a0 to calculate the result Only a limited subset of partitions is used.<\/p>\n

    In Apache spark narrow transformations groups as a stage. That process is mainly<\/span> known as pipelining. Pipelining is an implementation mechanism.<\/p>\n

    In this mechanism, multiple<\/span> instructions get overlapped in the execution process. The computer pipeline automatically<\/span> gets divided into stages.<\/p>\n

    b. Wide Transformations<\/h5>\n

    Wide transformations are the result of groupByKey (func) and reduceByKey (func). As data may reside in many partitions of the parent RDD. These are used to compute the records by data in the single partition.<\/p>\n

    Wide transformations may also know as shuffle transformations. Even they may or may not depend on a shuffle. Shuffling means redistributing data across partitions. In other words, shuffling is the process of data transfer between stages.<\/p>\n

    2. Actions<\/h4>\n

    An action is an operation, triggers execution of computations and RDD transformations. Also, returns the result back to the storage or its program. Transformation returns new RDDs and actions returns some other data types. Actions give non-RDD values to the RDD operations.<\/p>\n

    It\u00a0forces the evaluation of the transformation process need for the RDD they may call on. Since they actually need to produce output. An action instructs Spark to compute a result from a series of transformations.<\/p>\n

    Simply\u00a0put, an action evaluates the RDD lineage graph. RDD lineage (RDD operator graph or RDD dependency graph) is a graph of all the parent RDDs of an RDD. This graph is mainly made as a result of applying transformations to the RDD.<\/p>\n

    That creates a logical execution plan. Logical execution plan starts with the earliest RDDs and also ends with the RDD. Ultimately, that plan produces the result of the action which is only called to execute.<\/em><\/p>\n

    Actions are one of two ways to send data from executors to the driver. Executors are agents that are responsible for executing different tasks. While a driver coordinates execution of tasks.<\/p>\n

    Accordingly, action eliminates the laziness of RDDs and convert that laziness into motion.<\/p>\n

    Limitations<\/h3>\n

    \"RDD<\/a><\/p>\n

    1. No Input Optimization Engine<\/h4>\n

    There are several spark advance optimizers like catalyst optimizer and tungsten execution engine. RDDs are not able to use these optimizers. There is no way in RDD for automatic optimization. To optimize, we can optimize each RDD manually.<\/p>\n

    2. Not Enough Memory<\/h4>\n

    This is a sort of storage issue when we are unable to store RDD due to its lack of memory. The RDD degrades itself when there is not enough space to store spark RDD in-memory or on disk.<\/p>\n

    All the partitions that are already overflowing from RAM can be later on stored in the disk. Even so, that will provide the same level of performance. To overcome this issue, we need to increase the size of RAM or Disk.<\/p>\n

    3. Runtime type safety<\/h4>\n

    There is no run-time type safety in spark RDD spark RDD. Runtime type safety means it does not allow us to check error at the same time you are compiling the Program.<\/p>\n

    4. Handling Structured Data<\/h4>\n

    Spark RDD are user specific, they don\u2019t infer any schema of the ingested data like other APIs. Such as data frames as well as datasets. Moreover,\u00a0 when we need data immediately, a method to get and import data is data ingestion process.<\/p>\n

    Conclusion<\/h3>\n

    As a result, by analysis of above operations as well as features,\u00a0 RDD in\u00a0 Apache Spark increases the efficiency of our work. Also, make itself more useful for faster computations. Hence, these features enhance the performance of the system.<\/p>\n

    But there were some limitations of RDD. For example, no inbuilt optimization, storage and performance limitation etc.<\/p>\n

    Because of above-stated limitations, to make spark more versatile dataframes and datasets evolved.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Apache Spark RDD makes developer’s\u00a0\u00a0work more efficient, as it divides cluster into nodes to compute parallel operations on each node. Before anything else, we will go through the brief introduction of Spark RDD. After...<\/p>\n","protected":false},"author":1,"featured_media":73123,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[610,611,612,613,615],"class_list":["post-659","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-apache-spark-rdd-operations","tag-introduction-of-apache-spark-rdd","tag-limitations-and-operations-of-spark-rdd","tag-prominent-feature-of-spark-rdd","tag-transformation-action-with-example"],"yoast_head":"\nSpark RDD - Features, Limitations and Operations - TechVidvan<\/title>\n<meta name=\"description\" content=\"Apache Spark RDDs are a core abstraction of Spark which is immutable. In this blog, we will discuss a brief introduction of Spark RDD, RDD Features-Coarse-grained Operations, Lazy Evaluations, In-Memory, Partitioned, RDD operations- transformation & action RDD limitations & Operations.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Spark RDD - Features, Limitations and Operations - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Apache Spark RDDs are a core abstraction of Spark which is immutable. In this blog, we will discuss a brief introduction of Spark RDD, RDD Features-Coarse-grained Operations, Lazy Evaluations, In-Memory, Partitioned, RDD operations- transformation & action RDD limitations & Operations.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-05T12:26:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Features-Of-Spark-RDD-01-Copy.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Spark RDD - Features, Limitations and Operations - TechVidvan","description":"Apache Spark RDDs are a core abstraction of Spark which is immutable. In this blog, we will discuss a brief introduction of Spark RDD, RDD Features-Coarse-grained Operations, Lazy Evaluations, In-Memory, Partitioned, RDD operations- transformation & action RDD limitations & Operations.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/","og_locale":"en_US","og_type":"article","og_title":"Spark RDD - Features, Limitations and Operations - TechVidvan","og_description":"Apache Spark RDDs are a core abstraction of Spark which is immutable. In this blog, we will discuss a brief introduction of Spark RDD, RDD Features-Coarse-grained Operations, Lazy Evaluations, In-Memory, Partitioned, RDD operations- transformation & action RDD limitations & Operations.","og_url":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2018-01-05T12:26:29+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Features-Of-Spark-RDD-01-Copy.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Spark RDD – Features, Limitations and Operations","datePublished":"2018-01-05T12:26:29+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/"},"wordCount":1719,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Features-Of-Spark-RDD-01-Copy.jpg","keywords":["Apache Spark RDD Operations","Introduction of Apache Spark RDD","Limitations and Operations of Spark RDD","Prominent feature of Spark RDD","Transformation & Action with Example"],"articleSection":["Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/","url":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/","name":"Spark RDD - Features, Limitations and Operations - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Features-Of-Spark-RDD-01-Copy.jpg","datePublished":"2018-01-05T12:26:29+00:00","description":"Apache Spark RDDs are a core abstraction of Spark which is immutable. In this blog, we will discuss a brief introduction of Spark RDD, RDD Features-Coarse-grained Operations, Lazy Evaluations, In-Memory, Partitioned, RDD operations- transformation & action RDD limitations & Operations.","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Features-Of-Spark-RDD-01-Copy.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Features-Of-Spark-RDD-01-Copy.jpg","width":1200,"height":628,"caption":"Spark RDD Features"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/spark-rdd-features\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Spark RDD – Features, Limitations and Operations"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/659","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=659"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/659\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73123"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=659"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=659"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=659"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}