{"id":2023,"date":"2018-01-13T09:29:52","date_gmt":"2018-01-13T09:29:52","guid":{"rendered":"https:\/\/techvidvan.com\/tutorials\/?p=755"},"modified":"2018-01-13T09:29:52","modified_gmt":"2018-01-13T09:29:52","slug":"apache-spark-sql-datasets","status":"publish","type":"post","link":"https:\/\/techvidvan.com\/tutorials\/apache-spark-sql-datasets\/","title":{"rendered":"Introduction to Apache Spark SQL Datasets"},"content":{"rendered":"

Spark datasets is a distributed collection of data. It is a new interface, provides benefits of RDDs with Spark SQL\u2019s optimized execution engine. In this blog, we will learn the concept of Spark SQL dataSets.<\/p>\n

We will also focus on why datasets is needful, and what is the significance of encoder in the datasets? Moreover, we will cover the features of the SQL DataSets in Apache Spark.<\/p>\n

Also, understand how to create a SQL datasets in this spark tutorial.<\/p>\n

What is Spark SQL DataSet?<\/h3>\n

It is an interface, provides the advantages of RDDs with the comfort of Spark SQL\u2019s execution engine. It is a distributed collection of data. We can also create Spark datasets from JVM objects. By using functional transformations (map, flatMap, filter, etc.), it can be manipulated.<\/p>\n

Datasets API is only available in Scala and Java. R and Python do not have the support of the datasets API as Python is very dynamic in nature, it provides many of the benefits of the datasets API, such as we can access the field of a row by name naturally row.columnName.<\/p>\n

In addition, it is a strongly-typed immutable collection of objects, these are mapped to a relational schema. There is a new concept called an encoder<\/em>, at the core of the datasets API. However, an encoder is responsible for converting between JVM objects and tabular representation.<\/p>\n

By using Spark\u2019s tungsten binary format, all the tabular representation is stored. It allows operations on serialized data and also improves memory utilization. Datasets API is quite familiar, it provides many of the same functional transformations such as map, flatMap, filter.<\/p>\n

The encoder is a primary concept in serialization and deserialization framework. In Spark, there are many built-in encoders, which are very advance. Encoders generate bytecode to interact with off-heap data. Without a de-serializing entire object, the encoder provides on-demand access to individual attributes.<\/p>\n

It is structured as well as lazy query expression which triggers on the action, datasets represent logical plan internally. This plan tells the computational query, which we need to produce the data. That plan is also a base catalyst query plan for the logical operator to form a logical query plan.<\/p>\n

Afterwards, when we analyze this and resolve we can form a physical query plan. Most importantly,\u00a0 datasets reduce the memory usage.<\/p>\n

Why SQL DataSets in Spark?<\/h3>\n

As we know, there were some drawbacks with RDD as well as Spark dataframes, to overcome those limitations datasets introduced. As in dataFrame, data cannot be manipulated without knowing its structure.<\/p>\n

Since there was no clause for compile-time type safety. \u00a0While In RDD there was no automatic optimization. Thus, whenever required we need to do it manually, to overcome this issues we needed dataset.<\/p>\n

Datasets\u00a0inherit all the features of RDD and dataframe, such as:<\/p>\n