Apache Spark

Spark SQL isnumeric Function Alternative and Example

Most of the organizations are moving their data warehouse to the Hive and using Spark as an execution engine. Spark as an execution engine will boost the performance. In SQL, there are many options that you can use to deal with non-numeric values, for example, you can create user defined functions to filter out unwanted data. In this article, we will check Spark SQL isnumeric function alternative and examples. Spark SQL isnumeric Function Spark SQL, or Apache Hive does not provide support for is numeric function. You have to write…

Continue ReadingSpark SQL isnumeric Function Alternative and Example
Comments Off on Spark SQL isnumeric Function Alternative and Example
November 16, 2019
Apache Spark

Spark SQL DataFrame Self Join and Example

You can use Spark Dataset join operators to join multiple dataframes in Spark. Two or more dataFrames are joined to perform specific tasks such as getting common data from both dataFrames. In this article, we will check how to perform Spark SQL DataFrame self join using Pyspark. Spark SQL DataFrame Self Join using Pyspark Spark DataFrame supports various join types as mentioned in Spark Dataset join operators. A self join in a DataFrame is a join in which dataFrame is joined to itself. The self join is used to identify…

Continue ReadingSpark SQL DataFrame Self Join and Example
Comments Off on Spark SQL DataFrame Self Join and Example
November 14, 2019