{"id":2010,"date":"2018-01-06T12:36:02","date_gmt":"2018-01-06T12:36:02","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=655"},"modified":"2018-01-06T12:36:02","modified_gmt":"2018-01-06T12:36:02","slug":"persistence-and-caching-mechanism","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/persistence-and-caching-mechanism\/","title":{"rendered":"Persistence And Caching Mechanism In Apache Spark"},"content":{"rendered":"

In this article, we will learn about spark RDD persistence and caching\u00a0mechanism in detail. These are optimization techniques we use for spark computations. We will go through why do we need spark RDD persistence and caching, what are the benefits of RDD persistence in spark.<\/p>\n

We will also see what are the required storage levels to store persisted RDDs. Along with that, we will also study about, how to un-persist RDD in spark.<\/p>\n

\"RDD<\/a><\/p>\n

Understanding Persistence And Caching Mechanism in RDD<\/h3>\n

Spark RDD persistence and caching are optimization techniques. This may use for iterative as well as interactive Spark computations. Iterative computations<\/em> mean to reuse the results over multiple computations in multistage applications. Interactive computations<\/em> mean, allowing a two-way flow of information.<\/p>\n

These mechanisms help saving results for upcoming stages so, that we can use them. After these results, we can store RDD in memory and disk. Memory (most preferred) and disk (less Preferred because of its slow access speed). We can cache RDDs using cache ( ) operation. Similarly, we can also persist RDDs by persist ( ) operations.<\/p>\n

We can see Spark RDD persistence and caching one by one in detail:<\/p>\n

1. RDD Persistence Mechanism<\/h4>\n

As we know, RDDs are re-computable on each action by default due to its behavior. This phenomenon can be overcome by persisting the RDDs. So, that whenever we call an action on RDD, no re-computation takes place. When we call persist ( ) method, each computation stores the result in its partitions.<\/p>\n

To persist an RDD, we use persist ( ) method. We can use apache spark through scala, python, java etc coding. Persist( ) method will always store the data in JVM. In java virtual machine as an unserialized object, while working with java and scala.<\/p>\n

Similarly in python, calling persist() will serialize the data before persisting, serialize means (One-byte array per partition). There are options to store data in memory or disk combination is also possible.<\/p>\n

The actual persistence takes place during the first (1) action call on the spark RDD. Spark provides multiple storage options like memory or disk. That helps to persist the data as well as replication levels.<\/p>\n

When we apply persist method, RDDs as result can be stored in different storage levels. One thing to remember that we cannot change storage level from resulted RDD, once a level assigned to it already.<\/p>\n

2. Spark Cache Mechanism<\/h4>\n

Cache mechanism is one used to speed up the applications that access the same RDDs several times.<\/p>\n

Cache is a synonym of word persist or persist(MEMORY_ONLY), that signifies cache is nothing but persist with the default storage level MEMORY_ONLY.<\/p>\n

When to use caching<\/h5>\n

There are following situations in which we can use cache mechanism.<\/p>\n