{"id":2026,"date":"2018-01-11T11:00:24","date_gmt":"2018-01-11T11:00:24","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=840"},"modified":"2018-01-11T11:00:24","modified_gmt":"2018-01-11T11:00:24","slug":"apache-spark-paired-rdd","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/","title":{"rendered":"Apache Spark Paired RDD: Creation & Operations"},"content":{"rendered":"

In Apache Spark, Key-value<\/em> pairs are known as paired RDD<\/strong>. In this blog, we will learn what are paired RDDs in Spark in detail.<\/p>\n

To understand in deep, we will focus on following methods of creating spark paired RDD in and operations on paired RDDs in spark, such as transformations and actions in Spark RDD.<\/p>\n

The transformation such as groupByKey, reduceByKey, join, leftOuterJoin\/rightOuterJoin, while, actions like countByKey. But at first, we will learn brief introduction on RDDs in Spark.<\/p>\n

\"spark<\/a><\/p>\n

\u00a0Introduction – Spark RDDs<\/h3>\n

RDD<\/strong> refers to Resilient Distributed Datasets,\u00a0core abstraction and a fundamental data structure of Spark. RDDs in spark are immutable<\/em> as well as the distributed collection of objects. In RDD, each dataset is divided into logical partitions.<\/p>\n

That each partition may be computed on different nodes of the cluster. Spark RDDs can contain user-defined classes. Also, includes any type of Scala, python or java objects.<\/p>\n

It is a read-only, partitioned collection of records. Spark RDDs are the fault-tolerant collection of elements and it can be operated in parallel. There are generally three ways to create spark RDDs.<\/p>\n

Data in stable storage, other RDDs, and parallelizing existing collection in driver program. By using RDD, it is possible to achieve faster and efficient MapReduce operations.<\/p>\n

Introduction – Apache Spark Paired RDD
\n<\/b><\/h3>\n

Spark Paired RDDs are defined as the RDD containing a key-value pair. There is two linked data item in a key-value pair<\/strong> (KVP). We can say the key is the identifier<\/em>, while the value is the data corresponding<\/em> to the key value.<\/p>\n

In addition, \u00a0most of the Spark operations work on RDDs containing any type of objects. But on RDDs of key-value pairs, a few special operations are available. For example, distributed \u201cshuffle\u201d operations, such as grouping or aggregating the elements by a key.<\/p>\n

These operations are automatically available on RDDs containing Tuple2 objects, in Scala. In the Pair RDD functions class, the key-value pair operations are available. That wraps around an RDD of tuples.<\/p>\n

For example:<\/strong><\/p>\n

In this code we are using the reduceByKey operation on key-value pairs. We will count how many times each line of text occurs in a file:<\/p>\n

val lines1 = sc.textFile(\"data1.txt\")\nval pairs1 = lines.map(s => (s, 1))\nval counts1 = pairs.reduceByKey((a, b) => a + b)<\/pre>\n
There is one more method counts.sortByKey() <\/b>we can use.<\/p>\n
Importance of Apache Spark Paired RDD?<\/h3>\n
In many programs, pair RDDs of Apache Spark are a useful building block<\/em>. Operations that allow us to act on each key in parallel, it exposes those operations. Also, helps to regroup the data across the network.<\/p>\n
For instance, in spark paired RDDs reduceByKey() method aggregate data separately for each key and a join() method, which merges two RDDs together by grouping elements with the same key. It is very normal to extract fields from an RDD.<\/p>\n
For example, representing, for instance, an event time, customer ID, or another identifier. Also, use those fields in spark pair RDD operations as keys.<\/p>\n
Creating Paired RDD in Spark<\/h3>\n
By running a map() function that returns key or value pairs, we can create spark pair RDDs. On the basis of language, the procedure to build the key-value RDDs differs.<\/p>\n
    \n
  • \n
    In Python language<\/h4>\n<\/li>\n<\/ul>\n
    For the functions of keyed data to work, we need to return an RDD composed of tuples. Furthermore, for creating a pair RDD in spark using the first word as the key in Python programming language.<\/p>\n
    pairs = lines.map(lambda x: (x.split(\u201d \u201c)[0], x))<\/pre>\n
      \n
    • \n
      In Scala language
      \n<\/b><\/h4>\n<\/li>\n<\/ul>\n
      We also need to return tuples as shown in the previous example. Moreover, this will make functions on keyed data to be available. To provide the extra key or value functions, an implicit conversion on RDDs of tuples exists.<\/p>\n
      Furthermore, to create apache spark pair RDD, by using the first word as the keyword<\/p>\n
      val pairs = lines.map(x => (x.split(\u201d \u201c)(0), x))\n<\/pre>\n
        \n
      • \n
        In Java language
        \n<\/b><\/h4>\n<\/li>\n<\/ul>\n
        It doesn\u2019t have a built-in function of tuple function. So, by using the scala, only spark\u2019s java API has users create tuples.Tuple2 class. Although, users can construct a new tuple by writing new Tuple2(elem1, elem2) in java. Also, can access its relevant elements with the _1() and _2() methods.<\/p>\n
        In addition, when you are creating paired RDDs in Spark, we need to call special versions of spark\u2019s functions in java. For example, in place of the basic map() function the mapToPair () function should be used.<\/p>\n
        To create a Spark pair RDD, using the first word as the keyword<\/p>\n
        PairFunction<String, String, String> keyData =\n\nnew PairFunction<String, String, String>() {\n\npublic Tuple2<String, String> call(String x) {\n\nreturn new Tuple2(x.split(\u201d \u201c)[0], x);\n\n}\n\n};\n\nJavaPairRDD<String, String> pairs = lines.mapToPair(keyData);<\/pre>\n
        \u00a0Some Interesting Spark Paired RDD – Operations<\/h3>\n
        1.Transformation Operations
        \n<\/b><\/h4>\n
        All the transformations available to standard RDDs, Pair RDDs are allowed to use them. Even it can apply same rules from \u201cpassing functions to spark\u201d.<\/em><\/p>\n
        As there are tuples available in spark paired RDDs, we need to pass functions that operate on tuples, rather than on individual elements. Some of the transformation methods are listed here. For example:<\/p>\n
          \n
        • \n
          groupByKey<\/h5>\n<\/li>\n<\/ul>\n
          Basically, it groups all the values with the same key.<\/p>\n
          rdd.groupByKey()\n<\/pre>\n
            \n
          • \n
            reduceByKey(fun)<\/h5>\n<\/li>\n<\/ul>\n
            It uses to combine values with the same key.<\/p>\n
            add.reduceByKey( (x, y) => x + y)\n<\/pre>\n
              \n
            • \n
              combineByKey(createCombiner, mergeValue, mergeCombiners, partitioner)<\/h5>\n<\/li>\n<\/ul>\n
              By using a different result type, combine values with the same key.<\/p>\n
                \n
              • \n
                mapValues(func)<\/h5>\n<\/li>\n<\/ul>\n
                Without changing the key, apply a function to each value of a pair RDD of spark.<\/p>\n
                rdd.mapValues(x => x+1)\n<\/pre>\n
                  \n
                • \n
                  keys()<\/h5>\n<\/li>\n<\/ul>\n
                  Basically, Keys() returns a spark RDD of just the keys.<\/p>\n
                  rdd.keys()\n<\/pre>\n
                    \n
                  • \n
                    values()<\/h5>\n<\/li>\n<\/ul>\n
                    Generally, values() returns an RDD of just the values.<\/p>\n
                    rdd.values()<\/pre>\n
                      \n
                    • \n
                      sortByKey()<\/h5>\n<\/li>\n<\/ul>\n
                      Basically, sortByKey returns an RDD sorted by the key.<\/p>\n
                      rdd.sortByKey()<\/pre>\n
                      2. Action Operations
                      \n<\/b><\/h4>\n
                      Like transformations, actions available on spark pair RDDs are similar to base RDD. Basically, there are some additional actions available on pair RDDs of spark.\u00a0 Moreover, those leverages the advantage of the key\/value nature of the data. Some of them are listed below. For example,<\/p>\n
                        \n
                      • \n
                        countByKey()<\/h5>\n<\/li>\n<\/ul>\n
                        For each key, it helps to count the number of elements.<\/p>\n
                        rdd.countByKey()<\/pre>\n
                          \n
                        • \n
                          collectAsMap()<\/h5>\n<\/li>\n<\/ul>\n
                          Basically, it helps to collect the result as a map to provide easy lookup.<\/p>\n
                          rdd.collectAsMap()<\/pre>\n
                            \n
                          • \n
                            lookup(key)<\/h5>\n<\/li>\n<\/ul>\n
                            Basically, lookup(key) returns all values associated with the provided key.<\/p>\n
                            rdd.lookup()<\/pre>\n
                            Conclusion<\/h3>\n
                            Hence, \u00a0we have seen how to work with Spark key\/value data. Also, how to use the specialized functions and operations available in spark. Finally,\u00a0 we hope this article has given all your answers regarding spark paired RDDs.<\/p>\n","protected":false},"excerpt":{"rendered":"
                            In Apache Spark, Key-value pairs are known as paired RDD. In this blog, we will learn what are paired RDDs in Spark in detail. To understand in deep, we will focus on following methods...<\/p>\n","protected":false},"author":1,"featured_media":73282,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[839,840,841,842,843,844,845,846,847],"class_list":["post-2026","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-action-operation-in-spark","tag-apache-spark-paired-rdd-transformations-and-actions","tag-pair-rdds-transformations-and-actions","tag-paired-rdd-in-spark","tag-rdd-in-spark","tag-spark-rdd-operations","tag-spark-pairrddfunctions","tag-transformation-operation-in-spark","tag-working-with-keyvalue-pairs"],"yoast_head":"\nApache Spark Paired RDD: Creation & Operations - TechVidvan<\/title>\n<meta name=\"description\" content=\"Spark Paired RDD tutorial-introduction & importance of Spark Paired RDD, creation of paired RDD in spark and Transformation and action operations in RDD\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Spark Paired RDD: Creation & Operations - TechVidvan\" \/>\n<meta property=\"og:description\" content=\"Spark Paired RDD tutorial-introduction & importance of Spark Paired RDD, creation of paired RDD in spark and Transformation and action operations in RDD\" \/>\n<meta property=\"og:url\" content=\"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/\" \/>\n<meta property=\"og:site_name\" content=\"TechVidvan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TechVidvan\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-11T11:00:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-RDD.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"TechVidvan Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:site\" content=\"@vidvantech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"TechVidvan Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Spark Paired RDD: Creation & Operations - TechVidvan","description":"Spark Paired RDD tutorial-introduction & importance of Spark Paired RDD, creation of paired RDD in spark and Transformation and action operations in RDD","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/","og_locale":"en_US","og_type":"article","og_title":"Apache Spark Paired RDD: Creation & Operations - TechVidvan","og_description":"Spark Paired RDD tutorial-introduction & importance of Spark Paired RDD, creation of paired RDD in spark and Transformation and action operations in RDD","og_url":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/","og_site_name":"TechVidvan","article_publisher":"https:\/\/www.facebook.com\/TechVidvan\/","article_published_time":"2018-01-11T11:00:24+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-RDD.jpg","type":"image\/jpeg"}],"author":"TechVidvan Team","twitter_card":"summary_large_image","twitter_creator":"@vidvantech","twitter_site":"@vidvantech","twitter_misc":{"Written by":"TechVidvan Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#article","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/"},"author":{"name":"TechVidvan Team","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22"},"headline":"Apache Spark Paired RDD: Creation & Operations","datePublished":"2018-01-11T11:00:24+00:00","mainEntityOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/"},"wordCount":997,"commentCount":0,"publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-RDD.jpg","keywords":["action operation in spark","Apache Spark Paired RDD: Transformations and Actions","Pair RDDs: Transformations and Actions","Paired RDD in Spark","RDD in Spark","Spark RDD Operations","spark.PairRDDFunctions","Transformation operation in spark","Working with Key\/Value Pairs"],"articleSection":["Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/","url":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/","name":"Apache Spark Paired RDD: Creation & Operations - TechVidvan","isPartOf":{"@id":"https:\/\/techvidvan.com\/tutorials\/#website"},"primaryImageOfPage":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#primaryimage"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#primaryimage"},"thumbnailUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-RDD.jpg","datePublished":"2018-01-11T11:00:24+00:00","description":"Spark Paired RDD tutorial-introduction & importance of Spark Paired RDD, creation of paired RDD in spark and Transformation and action operations in RDD","breadcrumb":{"@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#primaryimage","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-RDD.jpg","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2019\/11\/Spark-RDD.jpg","width":1200,"height":628,"caption":"spark RDD"},{"@type":"BreadcrumbList","@id":"https:\/\/techvidvan.com\/tutorials\/apache-spark-paired-rdd\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/techvidvan.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Paired RDD: Creation & Operations"}]},{"@type":"WebSite","@id":"https:\/\/techvidvan.com\/tutorials\/#website","url":"https:\/\/techvidvan.com\/tutorials\/","name":"TechVidvan Blogs","description":"","publisher":{"@id":"https:\/\/techvidvan.com\/tutorials\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/techvidvan.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/techvidvan.com\/tutorials\/#organization","name":"TechVidvan","url":"https:\/\/techvidvan.com\/tutorials\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/","url":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","contentUrl":"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/2024\/03\/techvidvan-logo-200x50-1.webp","width":200,"height":50,"caption":"TechVidvan"},"image":{"@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TechVidvan\/","https:\/\/x.com\/vidvantech"]},{"@type":"Person","@id":"https:\/\/techvidvan.com\/tutorials\/#\/schema\/person\/e9c26e74dd3d87421f7ada9433b8cd22","name":"TechVidvan Team","description":"The TechVidvan Team delivers practical, beginner-friendly tutorials on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our experts are here to help you upskill and excel in today\u2019s tech industry."}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2026","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/comments?post=2026"}],"version-history":[{"count":0,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/posts\/2026\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media\/73282"}],"wp:attachment":[{"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/media?parent=2026"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/categories?post=2026"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techvidvan.com\/tutorials\/wp-json\/wp\/v2\/tags?post=2026"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}