The basic building block of an Apache Spark is RDD. The main abstraction Apache Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. In this article, we will check how to store the RDD using Pyspark Storagelevel. We will also check various storage levels with some examples. Pyspark Storagelevel Explanation Pyspark storagelevels are flags for controlling the storage of an resilient distributed dataset (RDD). Each StorageLevel helps Spark to decide whether to Use…

Apache Spark itself is a fast, distributed processing engine. As per the official documentation, Spark is 100x faster compared to traditional Map-Reduce processing. Another motivation of using Spark is the ease of use. You work with Apache Spark using any of your favorite programming language such as Scala, Java, Python, R, etc. In this article, we will check how to improve performance of iterative applications using Spark RDD cache and persist methods. Spark RDD Cache and Persist Spark RDD Caching or persistence are optimization techniques for iterative and interactive Spark applications. Caching and…

Apache Spark provides two kinds of operations: Transformations and Actions. We will check the commonly used basic Spark Transformations and Actions using pyspark. Create RDD from Local File You can use textFile spark context method to create RDD from local or HDFS file systems rdd = sc.textFile("file:////home/impadmin/test.txt") Related Articles: Apache Spark Architecture, Design and Overview Create RDD from HDFS File rdd = sc.textFile("hdfs:/localhost:8020/home/impadmin/test.txt") Basic Spark Transformations Transformations are Spark operation which will transform one RDD into another. Transformations will always create new RDD from original one. Below are some basic…