Persistence And Caching Mechanism In Apache Spark

Keeping you updated with latest technology trends, Join TechVidvan on Telegram

1.Objective – Persistence And Caching Mechanism

In this article, we will learn about spark RDD persistence and caching mechanism in detail. These are optimization techniques we use for spark computations. We will go through why do we need spark RDD persistence and caching, what are the benefits of RDD persistence in spark. We will also see what are the required storage levels to store persisted RDDs. Along with that, we will also study about, how to un-persist RDD in spark.

RDD Persistence And Caching Mechanism

2. Understanding Persistence And Caching Mechanism in RDD

Spark RDD persistence and caching are optimization techniques. This may use for iterative as well as interactive Spark computations. Iterative computations mean to reuse the results over multiple computations in multistage applications. Interactive computations mean, allowing a two-way flow of information. These mechanisms help saving results for upcoming stages so, that we can use them. After these results, we can store RDD in memory and disk. Memory (most preferred) and disk (less Preferred because of its slow access speed). We can cache RDDs using cache ( ) operation. Similarly, we can also persist RDDs by persist ( ) operations.

We can see Spark RDD persistence and caching one by one in detail:

2.1. RDD Persistence Mechanism

As we know, RDDs are re-computable on each action by default due to its behavior. This phenomenon can be overcome by persisting the RDDs. So, that whenever we call an action on RDD, no re-computation takes place. When we call persist ( ) method, each computation stores the result in their partitions.

To persist an RDD, we use persist ( ) method. We can use apache spark through scala, python, java etc coding. Persist( ) method will always store the data in JVM. In java virtual machine as an unserialized object, while working with java and scala. Similarly in python, calling persist() will serialize the data before persisting, serialize means (One-byte array per partition). There are options to store data in memory or disk combination is also possible.

The actual persistence takes place during the first (1) action call on the spark RDD. Spark provides multiple storage options like memory or disk. That helps to persist the data as well as replication levels. When we apply persist method, RDDs as result can be stored in different storage levels. One thing to remember that we cannot change storage level from resulted RDD, once a level assigned to it already.

2.2. Spark Cache Mechanism

Cache mechanism is one used to speed up the applications that access the same RDDs several times.

Cache is a synonym of word persist or persist(MEMORY_ONLY), that signifies cache is nothing but persist with the default storage level MEMORY_ONLY.

2.2.1. When to use caching

There are following situations in which we can use cache mechanism.

  • When we re-use RDD  while working in iterative machine learning applications
  • While we re-use RDD  in standalone spark applications
  • When RDD computations are expensive, we use caching mechanism. It helps in reducing the cost of recovery if, in case one executor fails.

2.3. Difference between Spark RDD Persistence and caching

This difference between the following operations is purely syntactic. There is the only difference between cache ( ) and persist ( ) method. When we apply cache ( ) method the resulted RDD can be stored only in default storage level, default storage level is MEMORY_ONLY. While we apply persist method, resulted RDDs are stored in different storage levels. As we discussed above, cache is a synonym of word persist or persist (MEMORY_ONLY), that means the cache is a persist method with the default storage level MEMORY_ONLY.

3. Persistence And Caching Mechanism – Need of Persistence Mechanism

It allows us to use same RDD multiple times in apache spark. As we know as many times we use RDD or we repeat RDD evaluation, we need to call action to execute. This process consumes much time as well as memory, while we perform iterative algorithm we require looking at data many times that time, that consumes ample of memory and time. To overcome this issue of repeated computation, these techniques of persistence introduced.

4. Persistence And Caching Mechanism – Benefits of RDD Persistence in Spark

Using techniques of RDD Persistence in apache spark has become beneficial. Listing following reasons below:

  • It enhances the speed of applications we perform generally. We can access same RDDs multiple times that increase the speed of our application.
  • They are very time efficient, before these methods of much time consumed in several processes. As this process comes in the picture, it reduces the time with an increase in work efficiency.
  • We were not able to use same RDDs, we have to afford many numbers of RDDs which became an expensive task for us. So, after persistence and caching is possible we can use same RDDs again and again. That results in reducing the cost and prove as cost-effective.
  • Likewise, we discussed earlier that RDD persistence is helping in reducing the time. This enhances the speed of application with less memory. It definitely lessens the execution time of the process.

5. Persistence And Caching Mechanism – Storage levels of Persisted RDDs

On applying persist method, RDDS takes place in respective storage levels. Those storage levels are:

 Persistence And Caching Mechanism - Storage levels of Persisted RDDs


5.1.  MEMORY_ONLY (Default level)

It is default memory, while it is must it stores data in available memory cluster (JVM), as an unserialized object. It also happens that if there is insufficient memory some of the data partitions may not be cached, that uncached data computed next time when we need it. In this option, we only use memory, not disk.


By this option, RDD stored as deserialized data objects. As sometimes RDD may not fit in the memory cluster, it stores the remaining part on the disk. Again we read the leftover part when needed. In this option, we use memory as well as disk.


In this option, RDD stored as serialized Java objects in memory. Serialized objects mean one-byte array per partition, this is much space efficient which saves memory. Due to this some data partitions may not be cached, so we only calculate remaining part as per requirement. In this option, we do not use the disk.


This option is as similar as MEMORY_ONLY_SER. Unlike, it saves the leftover part in the disk which is not stored in memory. This option uses both memory and disk storage.


This option stores RDD only on Disk. It makes only use of disk for storage purpose.

6. Persistence And Caching Mechanism – How to un-persist RDD in Spark

Cached data overreach the volume of memory, spark automatically expel the old data. This is actually a process named LRU, LRU refers to Last Recently Used. This algorithm categorizes the data as less used or frequently used. Either, it happens automatically or we can do it on our own by using the method calls un-persist, this is RDD.unpersist( ) method.

7. Persistence And Caching Mechanism –  Conclusion

Hence, Spark RDD persistence and caching mechanism are various optimization techniques, that help in storing the results of RDD evaluation techniques. These mechanisms help saving results for upcoming stages so that we can reuse it. After that, these results as RDD can be stored in memory and disk as well. To learn Apache Spark refer this books.

Reference – Apache Spark

If our blog is beneficial for you, don’t hesitate to write us.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.