BigData

Export Hive Table DDL, Syntax and Shell Script Example

There are many situations where you are required to export DDL's. For example, you are migrating some of your Hive tables to the RDBMS for reporting. If you are working as a Hadoop administrator, you should have knowledge on how to export Table DDL. In this article, we will check on how to export Hive table DDL to a text file using shell script and beeline connection string. Export Hive Table DDL As mentioned earlier, it is good to have a utility that allows you to generate DDL in Hive.…

Continue ReadingExport Hive Table DDL, Syntax and Shell Script Example
Comments Off on Export Hive Table DDL, Syntax and Shell Script Example
July 22, 2019
Data Warehouse

Database Table Denormalization Example

Bigdata technologies such as Hive, HBase, NoSQL taking over industry, thanks to its fast and distributed processing. Hadoop works on commodity hardware, so it is cheap too. Every organization wants to move its data to Bigdata world. If you are reading this article, your organization may be planning to migrate your relational database to Hadoop. Hadoop works best with denormalized tables. In this article, we will check how database Table denormalization works with an example. What is Table Denormalization? Before jumping into denormalization process, let us first understand what is…

Continue ReadingDatabase Table Denormalization Example
Comments Off on Database Table Denormalization Example
July 20, 2019
BigData

Impala Interval Data Type and Conversion Examples

Cloudera Impala Interval type is slightly different compared to Apache Hive interval data types. Only difference is it accept interval unit as a integer, where are in Hive it is string type. Interval type in Impala woks same way as in other relational databases such as Netezza, Vertica, Greenplum, Oracle, etc. In article, we will check more information on Impala interval data type and how to convert it. Impala Interval Data Type Impala interval type syntax accept unit specifications. The unit could be SECOND, HOUR, DAY, MONTH, YEAR. You have…

Continue ReadingImpala Interval Data Type and Conversion Examples
Comments Off on Impala Interval Data Type and Conversion Examples
July 19, 2019
BigData

Apache Hive Replace Function and Examples

By default, there is no Hive replace function available. String manipulation function replace is very much needed in case if you are manipulating strings and when there is a need to replace the particular value. Value could be junk value. In this article, we will check what are Hive replace function alternative methods that you can use whenever required. Hive Replace Function As mentioned earlier, Apache Hive does not provide support for replace function. However, it does provides support for regular expression functions and translate function. You can use any…

Continue ReadingApache Hive Replace Function and Examples
Comments Off on Apache Hive Replace Function and Examples
July 18, 2019
BigData

Hive Interval Data Types and Conversion Examples

Hive supports interval types in the same way as other relational databases such as Netezza, Vertica, Oracle, etc. It accepts interval syntax with unit specifications. You have to specify the units along withe interval value. For example, INTERVAL '1' DAY refers to day time. In this article, we will check Hive interval data types and its conversion examples. Hive Interval Data Types Hive version 1.2 and above supports interval types. Intervals of time units, Year to month intervals and Day to second intervals are available in hive version 1.2 and…

Continue ReadingHive Interval Data Types and Conversion Examples
Comments Off on Hive Interval Data Types and Conversion Examples
July 17, 2019
Apache Spark

Spark Modes of Operation and Deployment

Apache Spark Mode of operations or Deployment refers how Spark will run. Spark can run either in Local Mode or Cluster Mode. Local mode is used to test your application and cluster mode for production deployment. In this article, we will check the Spark Mode of operation and deployment. Spark Mode of Operation Apache Spark by default runs in Local Mode. Usually, local modes are used for developing applications and unit testing. Spark can be configured to run in Cluster Mode using YARN Cluster Manager. Currently, Spark supports Three Cluster…

Continue ReadingSpark Modes of Operation and Deployment
Comments Off on Spark Modes of Operation and Deployment
July 16, 2019
Apache Spark

Pass Functions to pyspark – Run Python Functions on Spark Cluster

Functions in any programming language are used to handle particular task and improve the readability of the overall code. By definition, a function is a block of organized, reusable code that is used to perform a single, related action. Functions provide better modularity for your application and a high degree of code reusing. In this article, we will check how to pass functions to pyspark driver program to execute on cluster. Pass Functions to pyspark Spark API require you to pass functions to driver program so that it will be…

Continue ReadingPass Functions to pyspark – Run Python Functions on Spark Cluster
Comments Off on Pass Functions to pyspark – Run Python Functions on Spark Cluster
July 11, 2019
Apache Spark

Pyspark Storagelevel and Explanation

The basic building block of an Apache Spark is RDD. The main abstraction Apache Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. In this article, we will check how to store the RDD using Pyspark Storagelevel. We will also check various storage levels with some examples. Pyspark Storagelevel Explanation Pyspark storagelevels are flags for controlling the storage of an resilient distributed dataset (RDD). Each StorageLevel helps Spark to decide whether to Use…

Continue ReadingPyspark Storagelevel and Explanation
Comments Off on Pyspark Storagelevel and Explanation
July 8, 2019
Apache Spark

Spark RDD Cache and Persist to Improve Performance

Apache Spark itself is a fast, distributed processing engine. As per the official documentation, Spark is 100x faster compared to traditional Map-Reduce processing. Another motivation of using Spark is the ease of use. You work with Apache Spark using any of your favorite programming language such as Scala, Java, Python, R, etc. In this article, we will check how to improve performance of iterative applications using Spark RDD cache and persist methods. Spark RDD Cache and Persist Spark RDD Caching or persistence are optimization techniques for iterative and interactive Spark applications. Caching and…

Continue ReadingSpark RDD Cache and Persist to Improve Performance
Comments Off on Spark RDD Cache and Persist to Improve Performance
July 8, 2019
Apache Spark

Spark SQL INSERT INTO Table VALUES issue and Alternatives

Spark SQL is gaining popularity because of is fast distributed framework. The Spark SQL is fast enough compared to Apache Hive. You can create tables in the Spark warehouse as explained in the Spark SQL introduction or connect to Hive metastore and work on the Hive tables. Not all the Hive syntax are supported in Spark SQL, one such syntax is Spark SQL INSERT INTO Table VALUES which is not supported. You cannot use INSERT INTO table VALUES option in spark. We will discuss the alternate approach with some examples.…

Continue ReadingSpark SQL INSERT INTO Table VALUES issue and Alternatives
Comments Off on Spark SQL INSERT INTO Table VALUES issue and Alternatives
July 7, 2019