BigData

How to Import Netezza Tables using Sqoop?

With growing data, organizations are moving computation part to Hadoop ecosystem. Apache Sqoop is an open source tool to import data from relational databases to Hadoop and vice versa. Apache Sqoop is one of the easiest tool to import relational database such as Netezza into Hadoop ecosystem. The Sqoop command allows you to import all tables, single table, execute query and store result in Hadoop HDFS. In this article, we will check how to import Netezza tables using Sqoop with some practical examples. Sqoop uses a connector based architecture which…

Comments Off on How to Import Netezza Tables using Sqoop?
March 7, 2019
Apache Spark

Steps to Connect Teradata Database from Spark – Examples

Apache Spark is one of the emerging bigdata technology, thanks to its fast and in memory distributed computation. You can analyze petabytes of data using the Apache Spark in memory distributed computation. You can connect Spark to all major databases in market such as Netezza, Oracle, etc. In this article, we will check one of methods to connect Teradata database from Spark program. You can connect using either Scala or Python Pyspark. For all examples in this article, we will use Scala to read Teradata tables. You can even execute…

Comments Off on Steps to Connect Teradata Database from Spark – Examples
March 5, 2019
Apache Spark

Steps to Connect Oracle Database from Spark – Examples

Apache Spark is one of the emerging bigdata technology, thanks to its fast and in memory distributed computation. You can analyze petabytes of data using the Apache Spark in memory distributed computation. In this article, we will check one of methods to connect Oracle database from Spark program. Preferably, we will use Scala to read Oracle tables. You can even execute queries and create Spark dataFrame. Steps to Connect Oracle Database from Spark Oracle database is one of the widely used databases in world. Almost all companies use Oracle as…

Comments Off on Steps to Connect Oracle Database from Spark – Examples
March 5, 2019
Apache Spark

How to Connect Netezza Server from Spark? – Example

I was working on one of the Spark project where we had a requirement to connect Netezza server from Spark. Integrating Netezza and Apache Spark enable analytic capabilities using the Apache Spark for data resides in Netezza database. There are various ways you can use to connect Netezza server from Spark program. You can also connect to the Netezza server from Pyspark. So, Why to Connect Netezza Server from Spark? This is an interesting question. To answer this, let us check how Apache Spark works. Apache Spark works on data…

Comments Off on How to Connect Netezza Server from Spark? – Example
March 4, 2019
Apache Spark

Basic Spark Transformations and Actions using pyspark

Apache Spark provides two kinds of operations: Transformations and Actions. We will check the commonly used basic Spark Transformations and Actions using pyspark. Create RDD from Local File You can use textFile spark context method to create RDD from local or HDFS file systems rdd = sc.textFile("file:////home/impadmin/test.txt") Related Articles: Apache Spark Architecture, Design and Overview Create RDD from HDFS File rdd = sc.textFile("hdfs:/localhost:8020/home/impadmin/test.txt") Basic Spark Transformations Transformations are Spark operation which will transform one RDD into another. Transformations will always create new RDD from original one. Below are some basic…

Comments Off on Basic Spark Transformations and Actions using pyspark
March 1, 2019
Apache Spark

Apache Spark SQL Introduction and Features

In an Apache Spark, Spark SQL is a module to work with structured and semi structured data. Any data that has schema is considered as structured data, for example, JSON, Hive tables, parquet file formats, etc. Whereas, semi structured data is something with no separation between the schema and the data. In this article, we will check Apache Spark SQL introduction and its features. Apache Spark SQL Introduction As mentioned earlier, Spark SQL is a module to work with structured and semi structured data. Spark SQL works well with huge…

Comments Off on Apache Spark SQL Introduction and Features
February 28, 2019
BigData

Apache Hive User-defined Functions

Apache Hive is a data warehouse framework on top of Hadoop ecosystem. The Apache Hive architecture is different compared to other Hadoop tools that are available. Being an open source project, Apache Hive has added a lot of functionalities since its inception. But it still lacks some basic functionalities that are available in traditional data warehouse systems such as Netezza, Teradata, Oracle, etc. In this post, we will check Apache Hive user-defined functions and how to use them to perform a specific task. Apache Hive User-defined Functions When you start…

Comments Off on Apache Hive User-defined Functions
February 27, 2019
BigData

Best Practices to Optimize Hive Query Performance

As we have seen in my other post Steps to Optimize SQL Query Performance, we can improve the performance of back-end SQL by adding simple improvement while writing SQL queries. Apache Hive architecture behaves differently with data and type of HQL query you write. In this post, we will check best practices to optimize Hive query performance with some examples. In data warehouse environment, we write lot of queries and pay very little attention to the optimization part. Tuning performance of Hive query is one of important step and require…

2 Comments
February 15, 2019
Netezza

Netezza Query History details using nz_query_history Table

Sometimes you may need to verify the queries that are running for a long time on production servers. There are several ways that you can perform this task. For instance, you can use Netezza administrative tool to verify long running queries. In this post, we will check how to get Netezza query history details using nz_query_history table. Netezza query history configuration steps are simple. You can follow below steps to use Netezza query history views to collect Netezza queries historical data in separate history table in optional history database. Why…

Comments Off on Netezza Query History details using nz_query_history Table
January 24, 2019
Apache Spark

Spark SQL Performance Tuning – Improve Spark SQL Performance

You can improve the performance of Spark SQL by making simple changes to the system parameters. It requires Spark knowledge and the type of file system that are used to tune your Spark SQL performance. In this article, we will check the Spark SQL performance tuning to improve Spark SQL performance. Related Article: Apache Spark SQL Introduction and Features Apache Spark Architecture, Design and Overview Data Storage Consideration for Spark Performance Before going into Spark SQL performance tuning, let us check some of data storage considerations for spark performance. Optimize…

Comments Off on Spark SQL Performance Tuning – Improve Spark SQL Performance
January 21, 2019