RDD is used for efficient work by a developer, it is a read-only partitioned collection of records. In this article. We will learn about the several ways to Create RDD in spark. There are following ways to Create RDD in Spark.<\/p>\n
Such as 1. Using parallelized collection 2. From existing Apache Spark RDD & 3. From external datasets. To get these concepts we will dive in, with few examples of the following methods to understand in depth.<\/p>\n
Spark RDD are core abstraction of apache spark. RDD refers to Resilient Distributed Datasets<\/strong>. Generally, we consider it as a technological arm of apache-spark, they are immutable in nature. It supports self-recovery, i.e. fault tolerance or resilient property of RDDs.<\/p>\n They are the logically partitioned collection of objects which are usually stored in-memory. RDDs can be operated on in-parallel. We can perform different operations on RDD as well as on data storage to form another RDDs from it.<\/p>\n There are two more ways to create RDD in spark manually by\u00a0cache<\/strong> and divide<\/strong> it manually. Users may also persist an RDD in memory. In parallel operation, we can reuse it efficiently. RDDs are a read-only partitioned collection of records.<\/p>\n As we cannot modify RDDs after once they created. This makes RDD to race different conditions and other failure scenarios.<\/p>\n There are two types of operations, we can perform on RDDs. They are transformations, which means to create a new dataset from the existing RDD.<\/p>\n Actions, return a value to the program after the completion of the computation on the dataset. Transformation returns new RDD, whereas action returns the new value to which are datatypes.<\/p>\n After learning about Apache Spark RDD, we will move forward towards the generation of RDD.<\/p>\n There are following ways to create RDD in Spark are:<\/p>\n 1.Using parallelized collection.<\/p>\n 2.From external datasets (Referencing a dataset in external storage system ).<\/p>\n 3.From existing apache spark RDDs.<\/p>\n Furthermore, we will learn all these ways to create RDD in detail.<\/p>\n RDDs can be created generally by the parallelizing method. It is possible by taking an existing collection from our driver program. Driver program such as Scala, Python, Java. Also by calling the sparkcontext\u2019s parallelize( ) method on it.<\/p>\n This is a basic method to create RDD which is applied at the very initial stage of spark. It creates RDD very quickly. It also initializes further operations on them at the same time. To operate this method, we need entire dataset on one machine.<\/p>\n Due to this property, this process is rarely used outside of testing and prototyping.<\/p>\n Considering the following example of sortbykey () method. In this programs, the values to be sorted is taken through the parallelized collection:<\/p>\n The number of partitions in which a dataset is cut into is a key point in the parallelized collection. we know spark cluster is logically partitioned. As we discussed earlier, we can also create RDD by its cache and divide it manually.<\/p>\n That means we can also set a number of partitions by our own. To set by own, we need to pass a number of partition as the second parameter in parallelize method.<\/p>\n So here we set the number of partitions 20 by our own.<\/p>\n We can see one more example below. In that, we are applying parallelize method and also giving the number of partitions by our own<\/p>\n So, this is an initial level method to create our own RDDs. It helps to create RDDs very quickly also.<\/p>\n This method is generally used to create datasets by users.<\/p>\n If any storage source supported by Hadoop, including our local file system it can create RDDs from it. Apache spark does support sequence files, textfiles, and any other Hadoop input format.<\/p>\n We can create textfile RDDs by sparkcontext\u2019s textfile method. This method uses the URL for the file (either a local path on the machine or database or a hdfs:\/\/, s3n:\/\/, etc URL). It also reads whole as a collection of lines.<\/p>\n Always be careful that the path of the local system and worker node should always be similar. The file should be available at the same place in the local file system and worker node.<\/p>\n We can copy the file of the worker nodes. We can also use a network mounted the shared file system.<\/p>\n To load a dataset from an external storage system, we can use data frame reader interface. External storage system such as file systems, key-value stores. It supports many file formats like:<\/p>\n In this example, we are providing a CSV file which returns dataset<Row> as a result.<\/p>\n Note<\/strong> \u2013 We have seen that in this example .rdd method is used. We use this format to convert Dataset <Row> to RDD <Row>.<\/p>\n In this example, we are providing a JSON file (one object per line ) which returns Dataset<Row> as a result.<\/p>\n In this example, we are providing a text file which returns Dataset of a string as a result.<\/p>\n As we discussed earlier, that RDD is immutable so, we can not change anything to it. So we can create different RDD from the existing RDDs. This process of creating another dataset from the existing ones means transformation.<\/p>\n As a result, transformation always produces new RDD. As they are immutable, no changes take place in it if once created. This property maintains the consistency over the cluster.<\/p>\n Some of the operations performed on RDD are map, \u00a0filter, count, distinct, flatmap etc.<\/p>\n In this example, we are providing a text file which returns Dataset of the string as a result.<\/p>\n Note <\/b>\u2013 In the above example RDD \u201cwordPair\u201d is created from existing RDD \u201cword\u201d using map ( ) transformation. This result contains word and starting character together of the same word.<\/p>\n Hence, we have learned all possible ways to\u00a0generate Spark RDD<\/em>\u00a0in-depth: parallelized collection, from external datasets and from existing Apache Spark RDD. As well as manually we can also create RDD by its cache<\/strong> and divide.<\/strong><\/p>\n So these datasets are no longer difficult for us to operate. This will enhance our efficiency while working on resilient distributed datasets.<\/p>\n","protected":false},"excerpt":{"rendered":" RDD is used for efficient work by a developer, it is a read-only partitioned collection of records. In this article. We will learn about the several ways to Create RDD in spark. There are...<\/p>\n","protected":false},"author":1,"featured_media":73353,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[663,664,665,666,667,658,668,669],"class_list":["post-2009","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-existing-apache-spark","tag-external-datasets","tag-how-to-create-spark-rdd","tag-parallelized-collection","tag-possible-ways-to-create-rdd-in-spark","tag-resilient-distributed-datasets","tag-tranformation-and-action","tag-ways-to-create-spark-rdd"],"yoast_head":"\n1. Using Parallelized collection<\/h4>\n
– For Example:<\/h4>\n
val\u00a0\u00a0\u00a0 data=spark.sparkContext.parallelize(Seq((\"sun\",01),(\"mon\",02),(\"tue\",03), (\"wed\",04),(\"thus\",05)))\n\nval sorted = data.sortByKey()\n\nsorted.foreach(println)\n<\/pre>\n
– For Example:<\/h4>\n
sc.parallelize(data, 20)<\/pre>\n
– For Example:<\/h4>\n
val rdd1 = spark.sparkContext.parallelize(Array(\"sun\",\"mon\",\"tue\",\"wed\",\"thu\",\"fri\"),4)\n\nval result = rdd1.coalesce(3)\n\nresult.foreach(println)\n<\/pre>\n
2. From external datasets (Referencing a dataset in external storage system)<\/h4>\n
a. CSV (String Path) Example<\/h5>\n
import org.apache.spark.sql.SparkSession\n\ndef main(args: Array[String]):Unit = {\n\nobject DataFormat {\n\nVal\u00a0\u00a0\u00a0 spark = \u00a0SparkSession.builder.appName(\"ExtDataEx1\").master(\"local\").getOrCreate()\n\nval dataRDD = spark.read.csv(\"path\/of\/csv\/file\").rdd<\/pre>\nb.\u00a0 json (String Path) Example<\/h5>\n
val dataRDD = spark.read.json(\"path\/of\/json\/file\").rdd<\/pre>\n
c. textfile (String Path) Example<\/h5>\n
val dataRDD = spark.read.textFile(\"path\/of\/text\/file\").rdd<\/pre>\n
3. From existing Apache Spark RDDs<\/h4>\n
– For Example:<\/h4>\n
val words=spark.sparkContext.parallelize(Seq(\"sun\", \"rises\", \"in\", \"the\", \"east\", \"and\", \"sets\", \"in\", \u201cthe\u201d, \"west\"))\n\nval wordPair = words.map(w => (w.charAt(0), w))\n\nwordPair.foreach(println)<\/pre>\n
Conclusion<\/h3>\n