As we know, there are so many distributed stream processing engines available. The question arises is why Apache spark streaming and what are its unique benefits. Spark Streaming tutorial totally aims at the topic \u201cSpark Streaming\u201d.<\/p>\n
To understand the topic better, we will start with basics of spark streaming, spark streaming examples and why it is needful in spark. Moreover, we will learn how streaming works in Spark, apache spark\u00a0streaming operations, sources of spark streaming.<\/p>\n
By the end, we will complete the topic with advantages of spark streaming over Big Data Hadoop & Storm.<\/p>\n
\u201cSpark Streaming<\/strong>\u201d is generally known as an extension of the core Spark API<\/em>. It is a unified engine that natively supports both batch and streaming workloads. Spark streaming enables scalability, high-throughput, fault-tolerant stream processing of live data streams.<\/p>\n It is a different system from others. Some engines either have streaming or have similar batch and streaming APIs, yet they compile internally to different engines.<\/p>\n Therefore, streaming came into the picture. This model offers both execution and unified programming for batch and streaming. There are some advantages over other traditional streaming systems. Majorly, there are 4 aspects of it.<\/p>\n Data ingestion is possible from many sources, such as Apache Flume, Kinesis, Kafka or TCP sockets. Processing is also possible by using complex algorithms expressed with high-level functions, such as map, reduce, join and window.<\/p>\n Afterwards, data which is processed can be pushed out to databases,\u00a0filesystems, and live dashboards. Although, can apply spark\u2019s machine learning & graph processing algorithms on data streams.<\/p>\n At first, input data streams enter further, it divides the data into batches. After that those batches processed through engine, to generate final results in batches.<\/p>\n DStream or discretized stream<\/i> is a high-level abstraction of spark streaming, that represents a continuous stream of data. Creation of DStreams is possible from input data streams, from following sources, such as Kafka, Flume, and Kinesis.<\/p>\n We can also create it by applying high-level operations on other DStreams. A DStream is just a sequence of spark RDDs, internally, we can write spark streaming programs in Scala, Java or Python.<\/p>\n There is traditional stream processing systems are designed. Those are integrated with a continuous operator model for data processing, working of the system is :<\/p>\n A set of worker nodes is available, that each runs one or more continuous operators. Continuous operator processes the streaming data one record at a time. Afterwards, it forwards the records to other operators in the pipeline.<\/p>\n By source operators data is received<\/span> from ingestion systems. By sink operators, it returns as output to downstream systems. Basically<\/span>, continuous operators are the natural model and very simple. Due to larger scale & complex real-time analytics, the traditional system met some challenges, such as:<\/p>\n To query the streaming data interactively or combining static datasets, it turns out to be very attractive, this was not easy in continuous operator systems. Those are not for new operators for ad-hoc queries. A single engine that can combine batch, streaming & interactive queries is necessary.<\/p>\n While working on complex workloads it always requires continuously<\/span> learning & updating data models. Also, querying the streaming data with SQL queries. Now, the developer\u2019s job much easier by having 1 abstraction across these analytic tasks.<\/p>\n It is must that system is able to recover from failures automatically for results. Static allocation of continuous operators to worker nodes is challenging in traditional systems.<\/p>\n Uneven allocation of processing load between workers causes bottlenecks in continuous operator system. The system must be able to adapt the resource allocation based on workload.<\/p>\n Earlier, as Hadoop have high latency that is not right for near real time processing needs<\/em>. In most cases, we use Hadoop for batch processing<\/em> while used Storm for stream processing<\/em>.<\/p>\n It leads to increase in code size, a number of bugs to fix, development effort, and causes other issues, which makes difference between Big data Hadoop and Apache Spark.<\/p>\n Ultimately, Spark Streaming fixed all those issues. It provides the scalable, efficient, resilient, and integrated system. This model offers both execution and unified programming for batch and streaming.<\/p>\n Although there is a major reason for its rapid adoption, is the unification of distinct data processing capabilities. It becomes hot cake for developers to use a single framework to attain<\/span> all the processing needs. In addition<\/span>, through Spark SQL streaming data can combine with static data sources.<\/p>\n In addition, it uses a new architecture called Discretized Streams<\/strong>, that offers rich libraries of Spark and fault tolerance property of the Spark engine.<\/p>\n As we know, continuous operator processes the streaming data one record at a time. Despite, processing one record at a time, it discretizes data into tiny, micro-batches. We can also say, spark streaming\u2019s receivers accept data in parallel. Also, buffer it in the memory of spark\u2019s worker’s nodes.<\/p>\n Afterwards, latency optimized engine runs small jobs to process batches and returns output. From the traditional continuous model, it differs as; computation is statically allocated to a node.<\/p>\n Tasks are assigned to the workers based on the locality of the data and available resources. It also enables better load balancing and faster fault recovery.<\/p>\n Moreover, a DStream is just a sequence of spark RDDs, abstraction fault-tolerant dataset. This offers, that we can process streaming data by using any spark library or code.<\/p>\n Computation of lost data in case of the node failure in traditional systems is not easy. It has to restart the failed operator on another node. As in spark, the computation is discretized<\/span> into small tasks, so that can run anywhere without affecting correctness.<\/p>\n Therefore<\/span>, failed tasks can spread on all nodes over cluster to perform recomputations. Hence, recover from the failure is faster than the traditional approach.<\/p>\n DStream is just a series of Spark RDDs, that allows batch, streaming workloads to interoperate seamlessly. We can apply Spark functions on each batch of streaming data, that can be interactively queried on demand. Since spark\u2019s worker memory stores it.<\/p>\n We can also integrate it with advanced processing libraries, such as SQL, machine learning, graph processing. RDDs, which are generated by DStreams can also convert into dataframes. Afterwards, they are queried with SQL.<\/p>\n Ability to batch data and leverage spark engine leads to almost higher throughput. Latencies of spark streaming are as low as a few hundred milliseconds.<\/p>\n Data stream converted into batches called DStreams, it is just a series of RDDs in spark that can process using Spark APIs. Afterwards, results are returned in batches. Spark streaming APIs are available in Scala, Java, and Python.<\/p>\n A state based on data coming in a stream called stateful computations, it offers window operations.<\/p>\n Input DStream is corresponding to a receiver object, that receives the data from a source and stores it in spark\u2019s memory for processing.<\/p>\n There are following two types of built-in streaming sources:<\/p>\n The sources those are directly available in the StreamingContext API, such as file systems, and socket connections.<\/p>\n Sources those are available through extra utility classes, such as Kafka, Flume, Kinesis, & many more. It also requires linking against extra dependencies.<\/p>\n There are two categories of a receiver, on the basis of \u00a0their reliability:<\/p>\n A receiver that sends an acknowledgment to a source, when they received the data. Also, stores in spark replication are a reliable receiver.<\/p>\n A receiver that does not send an acknowledgment to a source. We can use it as sources if, we do not want or need to go into the complexity of acknowledgment.<\/p>\n There are two types of operations, spark streaming supports:<\/p>\n Like RDDs, here also transformations allow modification of the data from input DStream. It also offers many transformations those are available on normal Spark RDD\u2019s. Few of them are:<\/p>\n Map(), repartition(numPartitions), filter(), flatMap(), union(otherStream),\u00a0 reduce(), count(), countByValue() and many more.<\/p>\n DStream\u2019s data can push out to external systems like a database, by output operations. Some of the following output operations are:<\/p>\n print(), saveAsTextFiles(prefix, [suffix])\u201dprefix-TIME_IN_MS[.suffix]\u201d, saveAsObjectFiles(prefix, [suffix]), and many more.<\/p>\n As a result, spark streaming overcome all issues with the traditional streaming systems. Now we can handle process batch and streaming workloads at once.<\/p>\n Hence, It becomes hot cake for developers to use a single framework to meet all the processing needs. Therefore, use of streaming also enhances system efficiency and performance.<\/p>\n","protected":false},"excerpt":{"rendered":" As we know, there are so many distributed stream processing engines available. The question arises is why Apache spark streaming and what are its unique benefits. Spark Streaming tutorial totally aims at the topic...<\/p>\n","protected":false},"author":1,"featured_media":73062,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[614],"tags":[860,861,862,863,864,648,865,866,867,868,869,870,871],"class_list":["post-771","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","tag-apache-spark-streaming","tag-architecture-of-spark-streaming","tag-diving-into-apache-spark-streamings-execution-model","tag-getting-started-with-spark-streaming","tag-operations-of-spark-streaming","tag-spark-streaming","tag-spark-streaming-examples","tag-spark-streaming-tutorial","tag-spark-streaming-a-quick-introductory-guide","tag-spark-streaming-what-is-it-and-whos-using-it","tag-streaming-in-spark","tag-what-is-spark-streaming","tag-working-of-spark-streaming"],"yoast_head":"\n\n
Internal Working<\/h4>\n
What is Spark Streaming<\/h3>\n
Why Streaming in Spark is needful?<\/h3>\n
\n
– Unification of Batch,\u00a0Streaming, and Interactive Workloads<\/h4>\n
– Advanced Analytics with Machine learning and SQL Queries<\/h4>\n
– Fast Failure and Straggler Recovery<\/h4>\n
– Load Balancing<\/h4>\n
Why Spark Streaming?<\/h3>\n
Architecture of Spark Streaming: Discretized Streams<\/h3>\n
Benefits of Discretized Stream Processing<\/h3>\n
1. Fast failure and straggler recovery<\/h4>\n
2. Unification of batch, streaming and interactive analytics<\/h4>\n
3. Advanced analytics like machine learning and interactive SQL<\/h4>\n
4. Performance<\/h4>\n
Working of Spark Streaming<\/h3>\n
Spark Streaming Sources<\/h3>\n
\n
Basic sources<\/h4>\n<\/li>\n<\/ul>\n
\n
Advanced sources<\/h4>\n<\/li>\n<\/ul>\n
\n
Reliable Receiver <\/b><\/h4>\n<\/li>\n<\/ul>\n
\n
Unreliable Receiver <\/b><\/h4>\n<\/li>\n<\/ul>\n
Spark Streaming Operations<\/h3>\n
1. Transformation Operations in Spark<\/h4>\n
2. Output Operations in Apache Spark<\/h4>\n
Conclusion<\/h3>\n